psychology: the Graduate Record Examination (GRE) Aptitude Tests and the GRE Advanced Test in Psychology; the Miller

Transcription

1 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 1969, 29, UNDERGRADUATE ABILITY FACTORS IN RELATIONSHIP TO GRADUATE PERFORMANCE 1 ALBERT MEHRABIAN University of California, Los Angeles THE present study investigated the relationships among criteria which can be employed in considering admission of students in graduate psychology programs. Further, the validity of these criteria for predicting graduate level performance was assessed. The following set of variables subsumes most of the criteria which are employed in considering candidates for graduate work in psychology: the Graduate Record Examination (GRE) Aptitude Tests and the GRE Advanced Test in Psychology; the Miller Analogies Test (MAT) ; undergraduate grade point average; grade point average in the junior and senior years of undergraduate work; increase in the grade point average in the last two years over the first two years of undergraduate work; number of mathematics and logic courses taken; rating of the department which the student attended as an undergraduate; sex of the student; amount of research experience of the student as an undergraduate; an average assessment of the overall promise of the student as a graduate student ; and a rating of the student s research in contrast to service orientation-the latter three being assessed from letters of recommendation and/or biographical statements of the student. 409 a series of There is research evidence relating scores on the MAT and the GRE, as well as undergraduate grade point average, to graduate school performance. The evidence relating to the MAT obtained in studies by Cureton, Cureton and Bishop (1949), Kelley and Fiske 1This study was supported by United States Public Health Service grant MH The author wishes to thank Linda Musun and Evelyn Hufnagel for their assistance in running the study, and the psychology staff at UCLA for their cooperation in providing the graduate student performance ratings.

2 410 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT ( 1951 ), Fahey (1953), Jensen ( 1953 ), Walters and Paterson ( 1953 ), and Platz, McClintock, and Katz (1959) provides support for the predictive validity of the MAT for graduate school performance in psychology. However, in one study conducted by Hymon (1957) the MAT did not significantly relate to either undergraduate or graduate grade point average or grades in research-oriented graduate courses. In one study employing the GRE tests, Capps and DeCosta (1957) found that of four variables-undergraduate grade point average, the two Aptitude scores from the GRE and the GRE Advanced Test scores-the GRE Advanced scores were the best predictors of graduate core course grades. Undergraduate grade point average has been found to relate to graduate grade point average and research potential. In an early study, Cureton, Cureton, and Bishop (1949) reported a.77 correlation between undergraduate grade point average and professors ratings of graduate students overall potential and promise in psychology. Capps and DeCosta (1957) found a 0.42 correlation between undergraduate GPA and graduate GPA. Also Platz, Mc- Clintock, and Katz stated that undergraduate GPA, as well as undergraduate GPA in science and mathematics, correlates significantly with graduate GPA. However, the latter, but not the former, also correlates with professor s ratings of the potential scientific contribution of graduate students. Finally, Jensen (1963) noted that undergraduate GPA is not always a significant predictor of first year graduate scholastic achievement, and that an objective test such as the Iowa Mathematics Aptitude Test tends to be a better predictor of scholastic achievement in graduate school. In sum, the evidence suggests that undergraduate grades and scores on content examinations, such as the GRE Advanced Test, tests of mathematical ability and grades in mathematics courses during undergraduate work, as well as tests of verbal ability such as in the verbal portion of the GRE or the MAT, can all partially predict the performance of graduate students in psychology. There does not seem to be any evidence, however, relating letter of recommendation ratings or research experience of undergraduates to their graduate school performance. The aims of the present study were two-fold: first, to characterize some of the ability factors based on the criteria employed in consid-

3 ALBERT MEHRABIAN 411 ering candidates for graduate school; and second, to obtain the relationships between the various admission criteria and graduate school performance in psychology. Thus, in the first part of the study, unselected applicants for the graduate psychology program at UCLA were all rated on each of the 13 admission criteria and these criteria were subsequently factor analyzed. In a second part of the study the admission criteria scores of psychology graduate students were all related to three indices of graduate psychology performance. The latter consisted of an average evaluation of the research competence and promise as a research worker of the student by those professors who had had contact with the student in a research setting; average grades of the student in content courses during his first year of graduate work; and average grades in first year statistics courses in graduate school. Method-Subjects. There were 266 subjects in the sample of candidates who had applied for admission to the graduate psychology program at UCLA. These cases were selected on the basis of availibility of ratings on all of the criteria noted below. The latter sample will be referred to as the &dquo;candidate&dquo; sample and will be distinguished from the following sample of graduate students. The graduate student sample consisted of 79 students presently enrolled in the psychology program at UCLA; their performance in graduate school was related to the admission criteria. Procedure. The following measures were obtained for each candidate based on information available from the candidate s admission file : scores on the Verbal and the Quantitative portions of the Aptitude Test of the Graduate Record Examination (GRE); score on the Graduate Record Examination Advanced Test in Psychology; score on the Miller Analogies Test (MAT); overall undergraduate grade point average (GPA); GPA during junior and senior years; increase in GPA in the last two, relative to the first two, years of undergraduate work; the number of mathematics and logic courses taken during undergraduate work; the rating of the graduate school faculty of the psychology department which the candidate attended as an undergraduate (Cartter, 1966, p. 56) ; research experience of the candidate; overall promise of the candidate as a research psychologist; and research versus service orientation of the candidate. The above measures were obtained as follows. In the case of the

4 - to 412 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT GRE and MAT scores, percentile scores were used. GPA scores were recorded on a 5-point scale ranging from zero to four. The rating of the school, taken from Cartter (1966), was translated into a numerical scale as follows: &dquo;distinguished&dquo; departments were assigned a score of 4; &dquo;strong&dquo; departments were assigned a score of 3; &dquo;good&dquo; departments were assigned a score of 2; and &dquo;adequate&dquo; departments were assigned a score of 1. For departments not included in the ratings provided by Cartter (1966), the rating of adequate, that is, a score of 1 was assigned. The research experience of a candidate was assessed on a 4-point scale including 0, for no research experience; 1, corresponding to one laboratory course; 2, corresponding to one year of supervised research experience; and 3, corresponding to two or more years of supervised research experience. The overall promise of a candidate was rated on a 10-point scale which ranged from 9: promises to graduate from graduate school with distinction and become a major research contributor in his area; 8: promises to graduate from graduate school with distinction and become a distinguished and active contributor in his area; 7: promises to graduate from graduate school with distinction and become a moderately active contributor in his area; 6: promises to graduate from graduate school with distinction and become a minor contributor in his area; 5: promises to graduate from graduate school with distinction but with uncertain potential as a research worker; 4: promises to graduate from graduate school with uncertain potential as a research worker; 3: promises to graduate from graduate school with no potential as a research worker; 2: will most probably get a doctorate; 1: will probably get a doctorate; and 0: is unlikely to get a doctorate. In addition to the candidate s research potential, his research orientation was assessed from his biographical statement as well as his letters of recommendation, with scores ranging from -3 corresponding to extreme service orientation +3 corresponding to extreme research orientation. For each candidate two additional indices were available from the action of the admissions committee who studied the folder of the candidate. The committee rated each candidate on a 6-point scale on the basis of an intuitive judgment of the information available for that candidate. A committee &dquo;evaluation&dquo; score was the mean rating based on independent judgments of at least two committee

5 ALBERT MEHRABIAN 413 members. The scale used was as follows. 0: not acceptable; 1: would barely make it through our program; 2: would probably get through our program but why knock ourselves out; 3: acceptable in terms of our present students; 4: desirable, would add to the department; 5: outstanding, accept immediately. A second index was the actual acceptance and non-acceptance letters mailed to the candidates. These were rated in the present study as follows. 0: student received a letter indicating that he was not accepted; 1: not accepted, placed on a waiting list but subsequently rejected; 2: not accepted, placed on a waiting list and subsequently accepted; 3: accepted; 4: accepted and given some indication of possible financial support; 5: accepted and offered possible financial aid, but the student decided to go elsewhere; 6: accepted and given definite promise of financial assistance; and 7: accepted and offered definite promise of financial assistance, but student chose to go to another school. In the second section of the study, our present graduate students were used as subjects. Their graduate school performance was related to the preceding criteria, and the following set of additional measures were also obtained: (1) an average rating of the student on the same 10-point scale reported above for assessing the overall research promise of a student-this scale was completed by all of those members of the faculty who had had extensive research contact with the student; (2) the average numerical grade of a student in all the content courses which he had taken in his first year; and finally (3) an average grade for all of the statistics courses which the student had taken his first year. Results. The 13 variables on which each subject in the sample was assessed were factor analyzed and a principal component solution was obtained. There were 6 factors with eigenvalues greater than unity, and these factors accounted for 75 per cent of the total variance. Varimax rotation of the first six factors yielded the following groupings of the criteria. For each factor reported below the criteria are listed in terms of decreasing loadings on the particular factor. GRE and MAT factor: this factor was defined solely by the GRE Verbal, the MAT, the GRE Advanced, and the GRE Quantitative scores. Grade point average factor: this was defined by the overall undergraduate GPA and GPA in the last two years of undergraduate work.

6 414 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT Research orientation factor: this was defined by the extent of research experience, a research in contrast to a service orientation, and finally by the letter of recommendation rating of the candidate regarding his potential as a graduate student and research worker in the field of psychology. Grade point average improvement factor: this was defined by the increase in the GPA of the student in the last two years over the first two years of undergraduate work. Sex factor: this was defined solely by the sex of the candidate. Mathematical training factor: this was defined by the total number of mathematics and logic courses which the student had taken and by a relatively low rating of the psychology program which the student attended as an undergraduate. An additional factor analysis was performed in which the evaluation rating of the admissions committee and the committee s acceptance decision were also included. The latter factor analysis yielded factors identical to the ones already reported. However, it was found that the ratings of the admissions committee as well as the acceptance decisions loaded on the first factor. In other words, this second factor analysis indicated that the scores on the GRE and the MAT mainly determined the rating of a student s promise as a graduate student in psychology by the admissions committee and their subsequent acceptance decisions. In a second set of analyses, the correlations of the admissions criteria with the three criteria of graduate school performance of each of the 79 graduate students were obtained. In reporting these correlations, the criteria of student sex, increase in the GPA of the last over the first two years of undergraduate work, the rating of the department which the student attended as an undergraduate, and his research experience are excluded, since these did not relate significantly to any of the three indices of graduate school performance. Table 1 contains the intercorrelations among the remaining criteria and correlations of these with the graduate school performance measures. With df 77, correlation coefficients exceeding =.222 are significant at the.05 level. Although some of the variables included in Table 1 do not significantly correlate with graduate school performance criteria, their inclusion in Table 1 provides some information about the differential effectiveness of the criteria in predicting graduate school per-

7 ALBERT MEHRABIAN 415 TABLE 1 Intercorrelation Matrix Relating Admission Criteria to Graduate Performance Criteria. r >.22, p <.05. formance. For example, the last two years GPA exhibits a stronger relationship to graduate school performance than overall GPA. Again, with the exception of the GRE Verbal index, all other GRE and MAT indices significantly relate to graduate school performance, with the GRE Advanced scores relating most consistently and strongly. One final analysis of the latter data was carried out as follows. In an attempt to compute a regression equation which can be used to estimate the overall quality of a student s graduate school and research performance promise, a stepwise regression analysis was performed on the following variables. The dependent variable was an overall index of graduate school performance which was the sum of the z-scores of a given subject s three indices of graduate school performance. Also, a single index for the GRE and MAT tests was computed for each student by simply summing his percentile scores on all four of the tests-henceforth referred to as the GRE-MAT index. The last two years GPA rather than the overall GPA was employed, since it exhibits a stronger relationship with the performance criteria. The remaining criteria were left unchanged and simply entered into the regression analysis. The results of the regression analysis for the first five indices entered into the equation were as follows : Graduate school performance = 1.34 (GRE-MAT index)

8 416 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT (letter of recommendation rating) (research orientation rating) + 18 (number of math and logic courses) + 91 (last two years GPA). Table 2 includes statistics for the various independent measures which were successively entered into the regression analysis. For each independent effect, the mean, the standard deviation, the range of possible scores, and the correlation with the overall graduate school performance index are reported. Furthermore, in the last column of that table, the multiple correlation coefficient with the dependent variable is reported for that step in the regression analysis. Thus, for instance, the coefficient equalled.507 when both the GRE-MAT and the letter of recommendation ratings were used in the equation. The equation reported above includes only the first five independent variables due to diminishing returns with the addition of the sixth and remaining variables. Discussion. The major thrust of the findings from the first sample of unselected candidates is that the Graduate Record Examination Aptitude and Advanced Tests, together with the Miller Analogies Test, define an ability factor which, incidentally, forms the basis for admissions judgments of the UCLA Psychology Faculty. Of the remaining ability factors for the unselected candidate sample, the TABLE 2 Statistics Relating to Variables Employed in the Stepwise Regression Analysis

9 ALBERT MEHRABIAN 417 candidate sex factor and the &dquo;increase in the last two years undergraduate GPA over the first two years GPA&dquo; exhibit the weakest relationships with graduate performance criteria and therefore would not seem to be as relevant for the characterization of candidates. Thus, in addition to the GRE-MAT index, there are three other candidate ability factors, namely, a grade point average, a research orientation and a mathematical training factor. The latter factor is defined by the number of mathematics and logic courses taken during undergraduate work as well as by a relatively low rating of the school which the candidate attended as an undergraduate. These suggest that candidates who have attended less distinguished psychology programs have had much more extensive coursework preparation at least in mathematics and logic than have applicants from the more distinguished departments, who are perhaps more representative of average students in undergraduate programs in those departments. Results from the second segment of the study help further clarify the four aforementioned ability factors, that is, the GRE-MAT index, the GPA, research orientation, and mathematical training factors. For example, the finding that the last two years GPA exhibits a stronger relation to the overall graduate school performance index than the overall GPA, suggests that the former be used as a summary index of GPA. The research orientation factor was defined in terms of letters of recommendation, research orientation, and research experience ratings. Of these three indices, the first two exhibit significant correlations with graduate performance and, according to the regression analysis, seem to be sufficiently different to merit separate treatment in attempts at the prediction of graduate school performance. (A stepwise regression analysis is such that at each successive stage the variable entered is the one which makes the greatest reduction in the error sum of squares. Thus, if there are two or three correlated variables, then one of these may be entered into the regression equation and serve essentially as a substitute for the remaining ones. However, because of the fact that the regression analysis yielded the letter of recommendation and the research versus service orientation ratings as two successive items in the equation, these two variables would seem to warrant separate treatment as indices of a student s promise as a research worker.) Finally, for the last, mathematical training, factor, the number of mathe-

10 418 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT matics and logic courses seems to suffice as an index of a candidate s ability defined by the factor. The above ability factors, which were obtained for an unselected sample of candidates, surprisingly enough continue to exhibit consistent relationships with graduate school performance even after extensive selection from the candidate sample which tends to make the graduate student sample a much more homogeneous one. Thus, considering the homogeneous quality of the sample of graduate students, the obtained relationship between the GRE-MAT index and the overall graduate school performance is quite impressive and would indicate that this index defines an ability factor that has validity in predicting graduate level performance. The next strongest predictor of graduate level performance is the letter of recommendation rating which may be more readily obtained from respondents by essentially presenting them with a scale such as the one employed in the present study. Respondents could also provide a service-research orientation rating for each candidate on a -3 to +3 scale such as the one used in this study. In sum, the findings of the present study are generally consistent with existing findings, but also provide information about additional criteria such as letter of recommendation ratings which are generally used in assessing candidate ability but whose validity has not been explored. Also, results of the regression analysis provide a basis for differential weighting of the various criteria in attempts at assessing a candidate s promise in graduate studies. REFERENCES Capps, Marian P. and DeCosta, F. Contributions of the Graduate Record Examination and the National Teacher Examinations to the Prediction of Graduate School Success. Journal of Educational Research, 1957, 50, Cartter, A. M. An Assessment of Quality in Graduate Education. Washington, D. C.: American Council on Education, Cureton, E. E., Cureton, Louise W., and Bishop, Ruth. Prediction of Success in Graduate Study in Psychology at the University of Texas. American Psychologist, 1949, 4, Fahey, G. L. Discriminatory Capacity of the University of Pittsburgh Examination among Graduate Students in Psychology. American Psychologist, 1953, 8, Hymon, S. R. The Miller Analogies Test and University of Pittsburgh Ph.D. s in Psychology. American Psychologist, 1957, 12, 35-6.

11 ALBERT MEHRABIAN 419 Jensen, R. E. Predicting Scholastic Achievement of First Year Graduate Students. EDUCATIONAL AND PSYCHOLOGICAL MEAS- UREMENT, 1953, 13, Kelley, E. L. and Fiske, D. W. The Prediction of Performance in Clinical Psychology. Ann Arbor: University of Michigan Press, Platz, A., McClintock, C., and Katz, D. Undergraduate Grades and the Miller Analogies Test as Predictors of Graduate Success. American Psychologist, 1959, 14, Walters, G. V. and Patterson, D. G. Miller Analogies Test Scores and Ratings of Ph.D. s in Psychology. American Psychologist, 1953, 8,