RELATIONSHIPS AMONG PROGRAM CHARACTERISTICS, INTERNSHIP PLACEMENT, AND EPPP SCORES IN DOCTORAL EDUCATION. A Thesis. Presented to the.

RELATIONSHIPS AMONG PROGRAM CHARACTERISTICS, INTERNSHIP PLACEMENT, AND EPPP SCORES IN DOCTORAL EDUCATION A Thesis Presented to the Faculty of San Diego State University In Partial Fulfillment of the Requirements for the Degree Master of Arts in Psychology by Lindsey Michelle Shamp Summer 2013

iii Copyright 2013 by Lindsey Michelle Shamp All Rights Reserved

iv ABSTRACT OF THE THESIS Relationships Among Program Characteristics, Intership Placement, and EPPP Scores in Doctoral Education by Lindsey Michelle Shamp Master of Arts in Psychology San Diego State University, 2013 Ultimately, the goal of students in health-directed psychology is to obtain licensure as a psychologist. In order to acquire licensure, education, training, and examinations are required. When applying to doctoral programs, particular attributes of those programs, such as data on education and training outcomes (IR-C20), are recommended to review. The implementing regulation section C20 ( IR-C20 ) is mandated by the American Psychological Association (APA) for public disclosure of training and education outcomes. This publicly reported data for each program includes the average time to completion, the program costs including assistantships and fellowships, the number and percentage of students who apply for and obtain internships (as well as characteristics of the internships students obtained), the attrition rate for eight years, and the number and percentage of graduates who have become licensed psychologists. Likewise, certain qualities of graduate programs, including area, type of degree, and overall competiveness, make doctoral candidates from those programs more competitive when applying for internships. An additional important factor to consider when assessing quality of training and education is the nature of the internship; in particular, the proportion of students who are placed in accredited positions. At this time, it is not known whether certain program characteristics are related to successful internship placement, and whether this in turn produces higher pass rates on the Examination for Professional Practice in Psychology (EPPP). The aim of this study is to measure these factors by simultaneously considering the effect of program characteristics, internship site placement, and pass rates of EPPP. Specifically, it was predicted that program characteristics, such as area (clinical, counseling, and school), degree type in clinical psychology (Ph.D. and Psy.D.), and membership to training councils, and IR-C20 data, will predict internship characteristics, such as accreditation status, which then will predict EPPP outcomes. An exhaustive search of all programs clinical, counseling, school and combined for full disclosure of IR-C20 data on their respective web sites was conducted and a sample of 360 programs was found. Hierarchical multiple regression analysis was performed to investigate EPPP scores amount of variance accounted by APA internship placement, after controlling for program characteristics such as admittance rate, licensure rate, time to completion, attrition, and Training characteristics such as area of psychology (Clinical, Counseling, or School), Degree (Ph.D. or Psy.D.), and membership to training councils. APA internship placement explained 36.9% of the variance in EPPP scores after controlling for all other independent variables (ΔR2=.369; F(8, 233) = 18.623; p <.001). Findings from this study indicate the potential value of investigating APA accredited internships in regards to the match rate imbalance. Future research should examine similar characteristics successful in licensure for the value of

standardizing requirements across programs. The results of this study will contribute to the growing body of knowledge on successful program attributes and their relationship to licensure. v

vi TABLE OF CONTENTS PAGE ABSTRACT... iv LIST OF TABLES... vii ACKNOWLEDGEMENTS... viii CHAPTER 1 MATCH RATE IMBALANCE OR ATRAINING AND EDUCATION PROBLEM...1 Method...6 Measures...6 Program Characteristics...6 Internship Characteristics...7 EPPP...8 Results...8 Discussion...16 REFERENCES...19

vii LIST OF TABLES PAGE Table 1. One Way ANOVA Table for EPPP Pass Rate...9 Table 2. Descriptive Statistics for EPPP Scores...10 Table 3. Tukey HSD Comparison for EPPP Scores...11 Table 4. One Way ANOVA Table for APA Internship...12 Table 5. Descriptive Statistics for APA Internship Placement...12 Table 6. Tukey HSD Comparison for APA Internship Placement...13 Table 7. Descriptive Statistics and Correlations for Dependent and Independent Variables...14 Table 8. Summary of Hierarchical Regression Analysis for Variables Predicting EPPP Scores...15

viii ACKNOWLEDGEMENTS I would like to express the deepest appreciation to my committee chair, Elizabeth Klonoff Ph.D., for her devotion and continual growing knowledge to the field of psychology and the advancement my of education and training. Without her guidance and persistent help this thesis would not be complete. I would also like to thank my committee as a whole for their time, dedication, and positive criticism. Thank you to my family, specifically my loving husband, for your many hours of support, encouragement, and dedication to my career.

1 CHAPTER 1 MATCH RATE IMBALANCE OR ATRAINING AND EDUCATION PROBLEM The goal in health services psychology, particularly in clinical, counseling, and school, is to obtain licensure as a psychologist. In order to practice applied psychology, a psychologist must be licensed, according to specific requirements set within each respective state (Association of State and Provincial Psychology Boards [ASPPB], 2012). This licensure is essential for protecting society from individuals inadequately trained. To obtain licensure an individual needs to complete three requirements, which include education, supervised experience (internship and in some cases postdoctoral work), and examination [Examination for Professional Practice in Psychology (EPPP)] (ASPPB, 2012). An internship is the final requirement of a student s education and training to earn their doctoral degree in psychology. This final year of education and training is intended to prepare students with supervised experience for licensure and ultimately a career in professional psychology (Collins, Callahan, & Klonoff, 2007). In order to obtain an internship, the majority of students apply, interview, and are matched to an internship site by the Association of Psychology Postdoctoral and Internship Centers (APPIC). Currently a large number of students who apply for internships ultimately fail to obtain a position through the match. The discrepancy between the number of students seeking internships and the number of internship slots is called the match rate imbalance. Over the last decade this has become a well-documented phenomenon (Baker, McCutcheon, & Keilin, 2007; Collins et al., 2007). Baker et al. (2007) noted that in a five-year span, students applying for internships had grown by 20%, but internship positions had only increased by 5%. Considering that the current APPIC match rate is 79%, this leaves 21% of doctoral students without a match to an internship site on the initial matching day (Association of Psychology Postdoctoral and Internship Centers [APPIC], 2010). Negative consequences of the inability to match to internship include overwhelming financial and emotional stress to the applicant

2 and increased time to complete degree, and lack of adequate training, such as unregulated, ad hoc, or marginal internship placements or lack of fundamental training resources and close supervision (Baker et al., 2007). In some states, if unsuccessful at completing an internship, students are unable to become licensed, leaving debt and training and education incomplete. Ultimately, consumers may receive a lower quality of care. The match rate imbalance has raised concerns about why some students match and others do not. One of the several approaches taken to scrutinize the internship problem has been assessing the quality of training and education doctoral students receive. In order to assess effective completion of education and training, one must look beyond characteristics that only produce successful match rates, and examine features that produce successful candidates for licensure, since ultimately this is the goal in health-directed psychology. When initially applying to doctoral programs, it is recommended that students look for qualities within a program that contribute to its overall competiveness such as admittance rate and other graduate outcomes. The American Psychological Association s (APA) Commission on Accreditation (CoA) requires all accredited programs to disclose specific education and training outcomes directly on their programs website (Commission on Accreditation [CoA], 2010). The publicly reported IR C-20 data for each program includes the average time to completion, the program costs including assistantships and fellowships, the number and percentage of students who apply for and obtain internships (as well as characteristics of the internships students obtained), the attrition rate for eight years, and the number and percentage of graduates who have become licensed psychologists. When applying to Ph.D. or Psy.D. programs, the IR C-20 data are thought to help students make informed decisions about which program to attend based on outcome characteristics of doctoral programs. For example, data taken from the Insiders guide to graduate programs in clinical and counseling psychology (2006/2007 ed.), found students receiving high rates of tuition reimbursement and an assistantship were more likely to receive APA accredited internships then students paying directly for their education (Graham & Kim, 2011). However, this study needs to be expanded as the data only assessed a one-year span. Additional doctoral program characteristics examined when addressing licensure outcomes include programs area (clinical and counseling), type of degree (Ph.D. and Psy.D.), and overall competiveness such as training focus. Studies have examined what

3 specific attributes make doctoral candidates more competitive when applying for internships (Callahan, Collins, & Klonoff, 2010; Graham & Kim, 2011). Graham and Kim (2011) compared area of Ph.D. received in psychology; students from clinical programs were significantly more likely to receive APA internships than those from counseling programs. While clinical and counseling psychologists receive the same licenses, the acceptance rates of counseling programs are higher than clinical Ph.D., but lower than Psy.D. programs (Norcross, Kohut, & Wicherski, 2005). Ph.D. programs, with a scientist-practitioner model of training, tend to focus more towards research while Psy.D., with a scholar-practitioner model programs, emphasize clinical training over training in research. When assessing degree obtained, Ph.D. programs received significantly higher rates of obtaining APA accredited internships than students attending Psy.D. programs (Graham & Kim, 2011). Another attribute important to consider is the accreditation status of internships in which programs place students. Callahan et al. (2010) examined characteristics of successfully matched interns and specific types of internship programs. They found an overall match rate of 85.2%, with 96.4% of these to an APA-accredited internship; no significant differences were found between Ph.D. and Psy.D. programs. This may be due to the study s focus on students from programs that are members of the Council of University Directors of Clinical Psychology (CUDCP), all of which represent the scientistpractitioner model of training within APA-accredited programs and hold a mission to promote advancement of training and education to further advance the field of psychology. The study also found CUDCP members, a subset of all the programs, obtained significantly higher match rates (85.2%) than the overall total of all programs reported by APPIC (76%) in 2009 (APPIC, 2010). Furthermore, the study found 96.4% of students matched received APA accredited internships. Although influences of program characteristics on graduate outcomes in general are well documented, much less is known regarding the influence of accreditation status from APA on internship and success with licensure. Guidelines were developed with the goal of providing an orderly way of assuring quality and accountability for education and training of psychologists. In general, accreditation provides a minimum standard of training protecting students, programs and society from those being trained inadequately (American Psychological Association Commission on Accreditation, 2009). While graduating from an

4 APA-accredited degree program is not required for licensure, in most states, or for obtaining a diploma, APA accreditation criteria have become the best available standards by which training programs and internship sites are judged (Oehlert, Sumerall, Lopez, & Merkley, 2002). It is a requirement of a doctoral student from an accredited program to successfully complete an internship. In early research Oehlert and Lopez (1998) noted that a common assumption from the majority of doctoral programs that hold APA-accreditation is that their students focus on sites that held APA accreditation and rarely seek non-accredited sites. Likewise, in a later study, Oehlert et al. (2002) specifically called for an examination of the quality of programs and internships receiving accreditation from APA or APPIC versus nonaccredited programs and internships. These studies questioned whether there is a lack of internships in general or whether there is a lack of availability of accredited internships. Keilin, Baker, McCutcheon, and Peranson (2007) found more than one third of unmatched applicants on the second phase consequently accepted a position that was not in an APPICmember program, and 20% accepted an unpaid internship. Not only is this a concern of quality, but also a student who has attended a non-accredited internship is limited in future occupational endeavors. Doctoral student completing non-accredited internships are excluded from employment at the Department of Veteran Affairs, military positions and prohibited for licensure in some states (Munsey, 2010). Professional psychology has permitted flexibility on requirements, such as lack of standardizing attendance of accredited programs as a condition of licensure, while other professions, medicine, dentistry, and pharmacy, consider this a minimum standard requirement (Belar, 2011). Similar to other professions, some have advocated accreditation should be the minimal standard for training programs (McCutcheon, 2011). While one perspective of standardizing of education and training is APA accreditation, an alternate perspective, such as those by members of California Psychology Internship Council (CAPIC), suggests other regional networks of internships are as effective in producing quality training as APA accreditation (Schaefer et al., 2011). While a large proportion of students obtain internship experience with out the use of APPIC matching system, or accreditation, these students continue to receive licensure. Since, APA does not review non-accredited establishments the quality of their training and education is undetermined.

5 Along with education and training through internship, examinations are required to obtain licensure. The Examination for Professional Practice in Psychology (EPPP) has been shown in research to be a successful predictor of profitable student graduate outcomes (Graham & Kim, 2011). EPPP, owned by the Association of State and Provincial Psychology Boards (ASPPB), is taken to obtain licensure to practice in professional psychology (Stedman & Schoenfeld, 2011). The examination covers eight content areas: biological, cognitive-affective, social and multicultural, growth and life-span development, assessment and diagnosis, treatment, intervention, and prevention, research methods and statistics, and ethical, legal, and professional issues. Few studies have examined predictors of passing the EPPP. In one of the few such studies, Graham and Kim (2011) studied factors influencing successful completion of the EPPP. These factors included function of specialty program (clinical versus counseling), degree type in clinical psychology (Ph.D. versus Psy.D.), and APA accreditation status. The study concluded students in clinical programs scored higher on the EPPP than counseling; programs within psychology departments performed better than those in the colleges of education; students with a Psy.D. degree preformed significantly lower than students with a Ph.D. on the EPPP; and students who attended programs with APA accreditation scored significantly higher than students in non-accredited programs. Examination of past research suggests that there may be relationships between these factors that may identify characteristics successful in improving match rates and outcomes of EPPP scores. However, there is no current documented research that has investigated the relationship among all these factors within the same study. The present study is undertaken to examine the degree to which program characteristics predict internship placement site characteristics which then predicts EPPP scores. Based on past research, it is hypothesized that students in clinical programs will be more likely to obtain APA accredited internships thus leading to higher pass rates of the EPPP than students in counseling or school programs. Programs with students seeking Ph.D. degrees will be more likely to obtain APA accredited internships thus leading to higher pass rates of the EPPP scores than students in programs seeking Psy.D. degrees. Students in programs belonging to scientist-oriented doctoral training groups will be more likely to receive APA accredited internships and obtain significantly higher EPPP scores than programs in practice-orientated groups and programs not affiliated with training organizations. Also, programs with enhanced graduate outcome

6 characteristics (e.g., lower time to completion, lower admittance rates, lower attrition rate, and high percentage of licensure) will have higher rates of obtaining APA accredited internships than programs with worse graduate outcomes characteristics (e.g., higher time to completion, higher program costs, lower rate of assistantships and fellowships, higher attrition rate, and low percentage of licensure). METHOD The initial sample consisted of the approximately 430 programs listed in the Association of State and Provincial Psychology Boards (ASPPB) EPPP score report where test takers reported obtaining their degrees (ASPPB, 2010). An exhaustive search was conducted between February 2011 to June 2011, of all listed programs for both program characteristics (e.g., area of training and degree) and IR-C20 data ( Student Admissions, Outcomes, and Other Data ) on their respective web sites. If full disclosure was not found on the program s web site then the graduate department was contacted through email. If still no reply was received, those programs were excluded. While virtually no programs replied, this left a sample of 360 programs that had with IR-C20 data. MEASURES Three characteristics measured for outcome data were program characteristics, internship characteristics, and EPPP. Program Characteristics Area and degree. Programs self-identified their substantive area of training. Clinical (60.5%, N= 240), counseling (20.4%, N= 81), school and combined (19.1%, N= 76) programs were located in both freestanding professional psychology schools and traditional academic universities, including counseling and school programs in colleges of education. If the program was combination of clinical, counseling, and school it was categorized as school psychology. A total of 323 programs (81.6%) awarded the Ph.D. and 73 (18.4%) the Psy.D. Training council membership. Membership list were taken from each individual training council s website. These training councils consisted of Academy of Psychological Clinical Science (APCS) 14.1% (N= 51), the Council of University Directors of Clinical Psychology (CUDCP) 30.7% (N= 111), Council of Counseling Psychology

7 Training Programs (CCPTC) 18.3% (N= 66), Council of Directors of School Psychology Programs CDSPP 17.7% (N= 64), and National Council of Schools and Programs of Professional Psychology (NCSPP) 19.1% (N= 69). Program belonging to both APCS and CUDCP were categorized as APCS (Academy of Psychological Clinical Science [APCS], 2011; Council of Counseling Psychology Training Programs [CCPTP], 2011; Council of Directors of School Psychology Programs [CDSPP], 2011; Council of University Directors of Clinical Psychology [CUDCP], 2011; National Council of Schools and Programs of Professional Psychology [NCSPP], 2011). Time to completion. For time to completion, the mean number of years that students have taken to complete the program from the time of first matriculation was taken from IR- C20 from their respective web sites of each individual program. Also, each program provides the percentage of students in each cohort completing the program in fewer than five years, five years, six years, seven years, and more than seven years (CoA, 2010). Attrition rate. Attrition is defined as the number of matriculated students who have left the program for any reason divided by the total number of students matriculated in the program and was taken from IR-C20 from their respective web sites of each individual program (CoA, 2010). When discrepancies occurred, the conservative estimate was documented. Licensure. Percentage of graduates who have become licensed psychologists in the eight-year period was taken from IR-C20 from their respective web sites of each individual program (CoA, 2010). Admit Rate. Admittance rate was obtained from The Graduate Study in Psychology by American Psychological association for years 2008-2010 (American Psychological Association [APA], 2009, 2010, 2011). The information was provided voluntarily from questionnaires sent to graduate departments in psychology. Internship Characteristics IR-C20 requires programs to provide data on students success in obtaining APA/CPA-accredited internships for the past seven years, and these data were used in the analyses. provided data.

EPPP The current study used publicly available archival data from the Association of State and Provincial Psychology Boards (ASPPB) on national licensing exam Examination of Professional Practice in Psychology (EPPP) for 2010 (ASPPB, 2010). When applying to take the examination, applicants self-report the doctoral program of attendance. The EPPP total pass rate and each of the subtest percent by content of each graduate program were evaluated for candidates who tested between January 2005 and December 2009. Data for individual programs were only available when there were 3 or more graduates who took the test between January 2005 and December 2009. ASPPB recommends a passing rate equal to a scaled score of 500. 8 RESULTS Six one-way ANOVAs, assessing first differences in EPPP scores and then proportion of APA-accredited internships obtained by substantive area, degree type, and membership in training council were performed, and these are presented in Tables 1 through 6. Students completing degrees in the area of clinical psychology were significantly more likely to obtain higher EPPP scores than students in counseling or school F(3, 393) = 9.571; p <.001) (Table 1). Tukey post-hoc comparisons of the three groups indicate that clinical programs (M = 85.91, 95% CI [84.05, 87.78]) gave significantly higher EPPP scores than counseling (M = 78.77, 95% CI [75.28, 82.26]) programs or School programs (M = 76.51, 95% CI [72.63, 80.4]) (Table 3). Comparisons between Counseling and School were not statistically significant at p <.05 (Table 3). Students completing degrees in the area of school psychology were also significantly less likely to APA internship placement than students in clinical or counseling (F(3, 343) = 19.273; p<.001) (Table 4). Post hoc test found no differences were found between clinical and counseling programs (Table 6). Students in Ph.D. programs were significantly more likely to obtain higher EPPP pass rate scores than students receiving Psy.D. degrees (F(1, 394) = 31.58; p <.001) (Table 1). Students in Ph.D. programs were also significantly more likely to obtain higher APA internship placement than students receiving Psy.D. degrees (F(1, 345) = 87.17; p <.001) (Table 4). Students attending programs with membership to APCS and CUDCP scored significantly higher EPPP scores (F(4, 356) = 33.644; p <.001) (Table 1) and highest rate obtaining APA internships (F(4,

9 Table 1. One Way ANOVA Table for EPPP Pass Rate Sum of Variables Squares df Mean Square F p Area Between Groups 6770.674 3 2256.891 9.571.001*** Within Groups 92671.341 393 235.805 Total 99442.015 396 Degree Between Groups 7356.581 1 7356.581 31.58.001*** Within Groups 91783.883 394 232.954 Total 99140.464 395 Training Between Groups 25076.639 4 6269.16 33.644.001*** Membership Within Groups 66335.55 356 186.336 Total 91412.19 360 Note. ***p<.001 320) = 62.137; p <.001) then programs with membership in CCPTC, CDSPP, and NCSPP (Table 4). Tukey post-hoc comparisons of the five groups indicate that APCS (M = 94.36, 95% CI [92.58, 96.14]) and CUDCP (M = 90.09, 95% CI [87.9, 92.28]) gave significantly higher EPPP scores than CCPTC (M = 79.38, 95% CI [75.42, 83.34]) CDSPP (M = 75.96, 95% CI [71.7, 80.21]) or NCSPP (M = 71.64, 95% CI [68.15, 75.123]) (Table 3). Post hoc test found no differences were found between APCS and CUDCP or between CCPTC and CDSPP (Table 3). Hierarchical multiple regression analysis was performed to investigate the amount of variance accounted for by APA internship placement rates in EPPP scores, after controlling for program characteristics such as admittance rate, licensure rate, time to completion, attrition, and training characteristics such as area of psychology (Clinical, Counseling, or School), degree (Ph.D. or Psy.D.), and training organization membership. Prior to conducing a hierarchical multiple regression, the relevant assumptions of this statistical analysis were tested. The independent variables were examined for multicollinearity and were found to be within established limits (tolerance all greater than.286; VIF less than 2.025); the correlations between these variables are presented in Table 7. All independent variables correlated with EPPP scores. In the first step of hierarchical multiple regression, three

10 Table 2. Descriptive Statistics for EPPP Scores EPPP Mean N SD Area Clinical 85.914 240 14.6368 Counseling 78.77 81 15.7791 School 76.512 76 17.0024 Total 82.657 397 15.8466 Degree Ph.D. 84.662 323 15.3688 Psy.D. 73.547 73 14.7795 Total 82.613 396 15.8426 Membership APCS 94.357 51 6.3318 CUDCP 90.089 111 11.6224 CCPTC 79.376 66 16.112 CDSPP 75.956 64 17.0308 NCSPP 71.636 69 14.5144 Total 82.701 361 15.935 Note. Means reflected are percentages. training characteristics were entered: area of psychology, degree and training membership. The program characteristics (admittance rate, licensure rate, time to completion, and attrition) were entered at step two, and APA internship placement at step three. The variables were entered in this order as it seemed chronologically relevant in doctoral student education and career path. The results of step 1 indicated this model was statistically significant (F(3, 225) = 36.36; p <.001) and accounted for 32.7% of variance in EPPP score (Table 8). Introducing the program characteristics at Step 2 the total variance explained by the model was 38.4% (F(7, 221) = 19.67; p <.001). The introduction of program characteristics explained additional 5.7% variance in EPPP scores, after controlling for area of psychology, degree and

11 Table 3. Tukey HSD Comparison for EPPP Scores Mean Differencee (I-J) Std. Error 95% Confidence Interval Lower Bound Upper Bound (I) Area (J) Area Clinical Counseling 7.1438*** 1.9721 2.504 11.783 School 9.4023*** 2.0201 4.65 14.155 Counseling Clinical -7.1438*** 1.9721-11.783-2.504 School 2.2585 2.4509-3.508 8.025 School Clinical -9.4023*** 2.0201-14.155-4.65 Counseling -2.2585 2.4509-8.025 3.508 (I) Training membership (J) Training membership APCS CUDCP 4.2677 2.3092-2.064 10.599 CCPTC 14.9811*** 2.545 8.003 21.959 CDSPP 18.4006*** 2.5623 11.375 25.426 NCSPP 22.7206*** 2.5207 15.809 29.632 CUDCP APCS -4.2677 2.3092-10.599 2.064 CCPTC 10.7134*** 2.1218 4.896 16.531 CDSPP 14.1329*** 2.1425 8.259 20.007 NCSPP 18.4530*** 2.0927 12.715 24.191 CCPTC APCS -14.9811*** 2.545-21.959-8.003 CUDCP -10.7134*** 2.1218-16.531-4.896 CDSPP 3.4195 2.3947-3.146 9.985 NCSPP 7.7395*** 2.3503 1.296 14.184 CDSPP APCS -18.4006*** 2.5623-25.426-11.375 CUDCP -14.1329*** 2.1425-20.007-8.259 CCPTC -3.4195 2.3947-9.985 3.146 NCSPP 4.32 2.369-2.175 10.815 NCSPP APCS -22.7206*** 2.5207-29.632-15.809 CUDCP -18.4530*** 2.0927-24.191-12.715 CCPTC -7.7395*** 2.3503-14.184-1.296 CDSPP -4.32 2.369-10.815 2.175 Note. *p <.05, **p <.01, ***p<.001 training membership (ΔR 2 =.057; F(7, 221) = 19.67; p <.001) (Table 8). No relationship was found on admittance rate or attrition rate to EPPP scores. Finally, the addition of APA internship placement to the regression model explained 41.2% of the variation in the model (F(8, 220) = 19.289; p <.001). APA internship placement explained 2.8% of the variance in EPPP scores after controlling for all other independent variables (ΔR 2 =.028; F(8, 220) =

12 Table 4. One Way ANOVA Table for APA Internship Sum of Variables Squares df Mean Square F p Area Between Groups 41124.351 3 13708.117 19.273.001*** Within Groups 243960.893 343 711.256 Total 285085.244 346 Degree Between Groups 57895.197 1 57895.197 87.917.001*** Within Groups 227190.047 345 658.522 Total 285085.244 346 Training Membership Between Groups 115089.597 4 28772.399 62.137.001*** Within Groups 148174.265 320 463.045 Total 263263.862 324 Note. ***p<.001 Table 5. Descriptive Statistics for APA Internship Placement APA Internship Mean N SD Area Clinical 75.2235 220 27.29588 Counseling 83.469 63 14.71965 School 51.4203 64 33.53317 Total 72.3303 347 28.70446 Degree Ph.D. 78.6488 280 24.22417 Psy.D. 45.9246 67 31.01074 Total 72.3303 347 28.70446 Membership APCS 92.0589 47 7.68863 CUDCP 86.3882 102 14.46212 CCPTC 82.7477 56 14.80342 CDSPP 52.0918 52 33.39899 NCSPP 45.4115 68 28.75206 Total 72.52 325 28.50514 Note. Means reflected are percentages.

13 Table 6. Tukey HSD Comparison for APA Internship Placement Mean Difference (I-J) 95% Confidence Interval Lower Upper Bound Bound Std. Error (I) Program (J) Area Clinical Counseling -8.2455 3.8324-17.2666 0.7756 School 23.80326*** 3.80906 14.8371 32.7694 Counseling Clinical 8.2455 3.8324-0.7756 17.2666 School 32.04876*** 4.75994 20.8443 43.2532 School Clinical -23.80326*** 3.80906-32.7694-14.8371 Counseling -32.04876*** 4.75994-43.2532-20.8443 (I) Training (J) Training membership membership APCS CUDCP 5.67074 3.79363-4.7367 16.0782 CCPTC 9.3112 4.25684-2.367 20.9894 CDSPP 39.96711*** 4.3309 28.0857 51.8485 NCSPP 46.64737*** 4.08185 35.4492 57.8455 CUDCP APCS -5.67074 3.79363-16.0782 4.7367 CCPTC 3.64046 3.57887-6.1778 13.4587 CDSPP 34.29638*** 3.66665 24.2373 44.3555 NCSPP 40.97663*** 3.36885 31.7345 50.2187 CCPTC APCS -9.3112 4.25684-20.9894 2.367 CUDCP -3.64046 3.57887-13.4587 6.1778 CDSPP 30.65591*** 4.14407 19.2871 42.0247 NCSPP 37.33616*** 3.88306 26.6834 47.9889 CDSPP APCS -39.96711*** 4.3309-51.8485-28.0857 CUDCP -34.29638*** 3.66665-44.3555-24.2373 CCPTC -30.65591*** 4.14407-42.0247-19.2871 NCSPP 6.68025 3.96411-4.1949 17.5554 NCSPP APCS -46.64737*** 4.08185-57.8455-35.4492 CUDCP -40.97663*** 3.36885-50.2187-31.7345 CCPTC -37.33616*** 3.88306-47.9889-26.6834 CDSPP -6.68025 3.96411-17.5554 4.1949 Note. *p <.05, **p <.01, ***p<.001

14 Table 7. Descriptive Statistics and Correlations for Dependent and Independent Variables APA internship EPPP Area Degree Training Membership Admit Licensure Rate Time to Completion Attrition EPPP 1 Area -0.276*** 1 Degree -0.375*** -0.123* 1 Training Membership -0.561*** 0.336*** 0.666*** 1 Admit -0.238*** -0.043 0.385*** 0.421*** 1 Licensure Rate 0.231*** -0.346*** 0.071-0.124-0.005 1 Time to Completion 0.374*** -0.078-0.429*** -0.409*** -0.256*** -0.155** 1 Attrition -0.143* 0.065 0.122* 0.2*** 0.193** -0.147* -0.051 1 APA internship 0.5*** -0.211*** -0.518*** -0.665*** -0.325*** 0.249*** 0.157** -0.159** 1 Means 82.66 19.45 66.27 6.19 10.49 72.33 Standard Deviations 15.85 16.58 21.27 0.94 7.84 28.7 Note. N= 229; *p <.05, **p <.01, ***p<.001

15 Table 8. Summary of Hierarchical Regression Analysis for Variables Predicting EPPP Scores Variable B SE β t R R² R² Step 1 0.521 0.3278*** Area -2.379 1.208 -.132* -1.97 Degree -3.055 3.059-0.084-0.999 Training Membership -4.773 0.92 -.461*** -5.189 Step 2 0.62 0.384*** 0.057*** Area -1.057 1.233-0.058-0.857 Degree -1.123 3.033-0.031-0.37 Training Membership -4.186 0.937 -.404*** -4.466 Admit 0.002 0.056 0.002 0.036 Licensure rate 0.153 0.046.195*** 3.351 Time to Completion 3.462 0.958.221*** 3.613 Attrition -0.027 0.097-0.015-0.273 Step 3 0.642 0.412*** 0.028*** Area -0.993 1.208-0.055-0.822 Degree 0.801 3.027 0.022 0.265 Training Membership -2.876 1.002 -.278** -2.871 Admit 0.014 0.055 0.015 0.258 Licensure rate 0.121 0.046.154** 2.634 Time to Completion 3.996 0.952.255*** 4.197 Attrition -0.028 0.095-0.016-0.293 APA internship 0.134 0.041.24*** 3.262 Note. N = 229; *p <.05, **p <.01, ***p<.001

16 19.289; p <.001) (Table 8). When independent variables were included in step three of the regression model, only licensure rate and time to completion contributed significantly to the EPPP scores over and above APA internship placement. The unstandardized regression coefficients (B) and intercept, the standardized regression coefficients (β), for the full model are reported in Table 8. DISCUSSION The purpose of this study was to examine the relationship between program characteristics, internship placement site characteristics, and EPPP scores. The hypothesis was partially supported as some program characteristics and placement of APA internships predicted scores on EPPP. Independent of other program characteristics, students in clinical Ph.D. programs were more likely to receive APA internships and performed higher on the EPPP than student in clinical Psy.D programs. Additionally, students in clinical Ph.D. programs were more likely to receive APA internships and performed higher on EPPP than student in other areas, while students in school Ph.D. programs were least likely to obtain APA internship and performed the lowest on EPPP. This is consistent with other findings (Graham & Kim, 2011). The hypothesis with membership organization training was supported, in that students in programs with membership to APCS and CUDCP were more likely to receive APA accredited internships and scored significantly higher EPPP scores than students in programs with memberships to CCPTC, CDSPP, and NCSPP. It is noteworthy those programs with membership to APCS and CUDCP have a tendency to be scientist-oriented, whereas programs with membership in NCSPP have a tendency to be practice-oriented. It has been reported that students training in scientist-oriented programs match at higher rates than students trained in more practice-oriented programs (Neimeyer, Rice, & Keilin, 2007). While not all program characteristics were found to be significant, (e.g., admittance rates and attrition rates), it is noteworthy those students who were unable complete a program was also unable to take EPPP. Therefore it is unknown how students would perform on EPPP in programs with high attrition rates. Students in programs receiving lower EPPP scores had lower time to completion and lower licensing rates. The current findings raises concerns about the quality some students are receiving in their training and

17 education as students more likely to receive non-accredited internships positions are also receiving lower EPPP scores. While the match rate imbalance is a current topic of interest in psychology, is the current problem then a match rate imbalance, or is the imbalance a symptom of a much larger problem? Hatcher (2011) likened the internship match imbalance to a common-pool resource (CPR) problem due to unrestricted open-access to any program to any student, using all resources available. Furthermore, this increase in use has lead to overuse resulting in lower quality-vetted positions. While suggestions have been proposed to assist with standardizing internship matches, none yet have been effective at decreasing the imbalance. Hatcher suggests management of the internship problem by a group, such as Council of Chairs of Training Councils (CCTC), to facilitate and govern the current process. CCTC is suggested, as it is voluntary group of leaders currently working together to solve the imbalance. CCTC membership includes training and education organizations and agencies providing accreditation or credentials. Callahan et al. (2010) focused on changes to improve quality by proposing multiple governing bodies working in collaboration to limit the internship match to doctoral students attending APA-accredited programs, limit the number of accredited programs per educational setting, and require accredited programs to fund all students. Even with widespread efforts focused on solutions to reduce this dire problem, the match rate imbalance continues to increase at significant rates. Many calls for change have be proposed, however such measures have been unsuccessful as implementation can be costly or time consuming. There are limitations to this study. The first limitation is the lack of information given by programs not required to report IR-C20 data. The number of programs with students taking the EPPP is over 500 programs however, only 360 were found reporting IR-C20 data. The programs not accounted for in this study, which are themselves no accredited, could possibly have higher rate of unaccredited internship positions, as students attending APA accredited programs are likely to attend APA internships. Due to many programs noncompliance with providing cost of attendance information, another limitation was inability to assess the relationship between this variable and EPPP scores and accredited internship rates. APA has recently increased enforcement of the requirements to report IR-C20 data, which would allow the missing information needed to be available. Future research is needed to

18 examine these variables, as they may be significant factor to success on internships and EPPP scores. Another limitation of the current study is the lack of information of the individual student such as demographics, GRE scores, and GPA. Students unable to receive positions at accredited doctoral programs, due to lower GRE score and lower GPA, may find unaccredited programs more appealing. It may be more convenient for students to attain positions at unaccredited programs, as these programs have higher acceptance rates and accredited programs can be more competitive in accepting students. Students attending these programs are also less competitive when applying for internships and subsequently receive lower EPPP scores. This would then be due to the characteristics of the individual student rather than the characteristics of the program or internship. Collins, Callahan, and Klonoff (2007) suggest stairway model for competency suggesting model focus on three major steps: Trainee Characteristics, Doctoral Program Opportunities, and the Internship Year. Evaluating individual student characteristics, such as undergraduate environment or personal goals would be relevant as student characteristics would be similar within each program. Findings from this study indicate the potential value of investigating not only the type of internship placement that students are receiving, but also the characteristics of internship applicants as well as the characteristics of doctoral program which high success rates of student to internship matches. In the end, APA accreditation for internships in professional psychology is fundamental for the preparation for practice as a psychologist. Ultimately, accreditation sets standardization for a minimum quality among programs and internships. Allowing doctoral students to attend such internships that are unstandardized fosters the lack of quality in internships and training standards. All of these outcomes can have a long-term effect on the public welfare and the reputation of professional psychology. The interplay between different perspectives of the current problem assisted in gaining a more comprehensive understanding of the match rate, because any one perspective consistently offers only a partial account of a complex phenomenon. However, it is imperative for the field to bring focus to address training and education issues such as the match rate imbalance and more specifically accreditation as standardization for a minimum quality among programs and internships.

19 REFERENCES Academy of Psychological Clinical Science [APCS]. (2011). 2011 institutional membership listing. Retrieved from http://acadpsychclinicalscience.org/index.php? page=members American Psychological Association [APA]. (2009). The graduate study in psychology 2009. Washington, D.C.: American Psychological Association. American Psychological Association [APA]. (2010). The graduate study in psychology 2010. Washington, D.C.: American Psychological Association. American Psychological Association [APA]. (2011). The graduate study in psychology 2011. Washington, D.C.: American Psychological Association. American Psychological Association Commission on Accreditation. (2009). Guidelines and principles for accreditation of programs in professional psychology. Washington, D.C.: American Psychological Association Office of Program Consultation and Accreditation. Association of Psychology Postdoctoral and Internship Centers [APPIC]. (2010). 2010 APPIC match statistics. Retrieved from http://www.appic.org/match/5_2_2_match _about_statistics.html Association of State and Provincial Psychology Boards [ASPPB]. (2010). Examination of professional practice in psychology (EPPP). 2010 psychology licensing exam scores by doctoral program. Retrieved from http://www.asppb.net/i4a/pages/index. cfm?pageid=3571 Association of State and Provincial Psychology Boards [ASPPB]. (2012). Requirements for licensure or registration to practice psychology. Retrieved from http://www.asppb.net/i4a/pages/index.cfm?pageid=3495 Baker, J., McCutcheon, S., & Keilin, W. (2007). The internship supply-demand imbalance: The APPIC perspective. Training and Education in Professional Psychology, 1(4), 287-293. Belar, C. D. (2011). A developmental milestone. Monitor on Psychology, 42, 49. Callahan, J. L., Collins, F. R., & Klonoff, E. A. (2010). An examination of applicant characteristics of successfully matched interns: Is the glass half full or half empty and leaking miserably? Journal of Clinical Psychology, 66(1), 1-16. Collins, F. R., Callahan, J. L., & Klonoff, E. A. (2007). A scientist-practitioner perspective of the internship match imbalance: The stairway to competence. Training and Education in Professional Psychology, 1(4), 267-275. Commission on Accreditation [CoA]. (2010). Commission on accreditation implementing regulations: Section C: IRs related to the guidelines and principals. Retrieved from http://www.apa.org/ed/accreditation/about/policies/implementing-guidelines.pdf

Council of Counseling Psychology Training Programs [CCPTP]. (2011). 2011 institutional membership listing. Retrieved from http://www.ccptp.org/membership/ directoryinstitutional.html Council of Directors of School Psychology Programs [CDSPP]. (2011). 2011 institutional membership listing. Retrieved from http://sites.google.com/site/cdspphome/cdsppmember-programs Council of University Directors of Clinical Psychology [CUDCP]. (2011). 2011 institutional membership listing. Retrieved from http://www.cudcp.us/ Graham, J. M., & Kim, Y. (2011). Predictors of doctoral student success in professional psychology: Characteristics of students, programs, and universities. Journal of Clinical Psychology, 67(4), 340-354. Hatcher, R. L. (2011). The internship supply as a common-pool resource: A pathway to managing the imbalance problem. Training and Education in Professional Psychology, 5(3), 126-140. Keilin, W., Baker, J., McCutcheon, S., & Peranson, E. (2007). A growing bottleneck: The internship supply-demand imbalance in 2007 and its impact on psychology training. Training and Education in Professional Psychology, 1(4), 229-237. McCutcheon S. (2011) The internship crisis: An uncommon urgency to build a common solution. Training and Education in Professional Psychology, 5(3), 144-148. Munsey, C. (2010). What would an unaccredited internship mean for your future? Retrieved from http://www.apa.org/gradpsych/2010/11/unaccredited.aspx National Council of Schools and Programs of Professional Psychology [NCSPP]. (2011). 2011 institutional membership listing. Retrieved from http://www.ncspp.info/ schools.htm Neimeyer, G. J., Rice, K. G., & Keilin, W. G. (2007). Does the model matter? The relationship between science-practice emphasis in clinical psychology programs and the internship match. Training and Education in Professional Psychology, 1(3), 153 162. Norcross, J. C., Kohut, J. L., & Wicherski, M. (2005). Graduate study in psychology, 1971 to 2004. American Psychologist, 60, 959 975. Oehlert, M. E., & Lopez, S. J. (1998). APA-accredited internships: An examination of the supply and demand issue. Professional Psychology: Research and Practice, 29(2), 189-194. Oehlert, M. E., Sumerall, S. W., Lopez, S. J., & Merkley, K. B. (2002). Internship placement data: Is nonplacement the only concern. Journal of Clinical Psychology, 58(10), 1323-1326. Schaefer, M. R., Newman, G. H., Perl, R., Morrison, A., Jordan, V. B., Wong, J.,... Montenegro, H. (2011). Shifting the paradigm: Alternative perspectives and solutions to increasing the availability of quality internships. Training and Education In Professional Psychology, 5(4), 209-212. 20

Stedman, J. M., & Schoenfeld, L. S. (2011). Knowledge competence in clinical and counseling training and readiness for internship. Journal of Clinical Psychology, 67(1), 1-5. 21