GRADUATE RECORD EXAMINATIONS PROGRAM

Transcription

1 VALIDITY and the GRADUATE RECORD EXAMINATIONS PROGRAM BY WARREN W. WILLINGHAM EDUCATIONAL TESTING SERVICE, PRINCETON, NEW JERSEY

2 Vaidity and the Graduate Record Examinations Program

3 Vaidity and the Graduate Record Examinations Program by Warren W. Wiingham Educationa Testing Service, Princeton, New Jersey

4 Copyright by Educationa Testing Service. A rights reserved.

5 FOREWORD In recognition of the importance of questions of vaidity to the Graduate Record Examinations program, the GRE Board asked Dr. Warren W. Wiingham of the staff of Educationa Testing Service to prepare a paper on the subject of vaidity and the GRE that coud provide a basis for further Board discussion and decisions. After reviewing Dr. Wiingham s paper, members of the Board agreed that it was an exceent document, which might we be of interest to others concerned with graduate admissions and the transition from undergraduate to graduate study. Accordingy, the GRE Board asked that the paper be pubished, and we are peased to make it avaiabe. Richard H. Armitage Chairman, GRE Board

6 INTRODUCTION... 1 BACKGROUND OF THE PROBLEM Nature of the GRE program... 2 Previousresearch... 3 The current importance of research on vaidity... 4 THE SCOPE AND MEANING OF VALIDITY OF THE GRE... 7 Conventiona interpretations of vaidity... 7 Socia interpretations of vaidity... 8 Construct vaidity of the program SIX PROPOSED OBJECTIVES FOR RESEARCH ON VALIDITY.. 11 Objective I: To encourage and faciitate institutiona vaidity studies Objective II: To dea effectivey with methodoogica issues concerning vaidity that require the GRE program s initiative Objective III: To deveop improved criteria of success in graduate study Objective IV: Popuation vaidity: How to improve it and enhance understanding of it Objective V: To increase institutiona use of summary programdata Objective VI: To systematicay insure the vaidity of revised or new measures resuting from programrenewa STATUS OF RESEARCH RELEVANT TO THE SIXOBJECTIVES Objective I: To encourage and faciitate institutiona vaidity studies Objective II: To dea effectivey with methodoogica issues concerning vaidity that require the GRE program s initiative Objective III: To deveop improved criteria of success in graduate study Objective IV: Popuation vaidity: How to improve it and enhance understanding of it Objective V: To increase institutiona use of summary program data Objective VI: To systematicay insure the vaidity of revised or new measures resuting from programrenewa REFERENCES AND BIBLIOGRAPHY

7 Vaidity is necessariy a major concern of any testing program. It is in the interest of the user that a test measure what it is supposed to measure and that it bear a reasonabe reationship to the criteria it is intended to predict. It is the responsibiity of a test sponsor to insure that these quaities prevai in the testing program. This principe of responsibiity appies with a specia force to a nationa program such as the Graduate Record Examinations (GRE), which affects arge numbers of peope. The stakes are high for both the individua and society. Because of these considerations, the vaidity of examinations sponsored by the GRE program has aways received cose attention. It is suggested in this paper, however, that current issues concerning their vaidity are critica to the immediate future of graduate admissions. Therefore, it is aso sug gested that research on vaidity shoud have high priority in the GRE program over the next severa years. The main purpose of this paper is to faciitate discussion of important issues concerning vaidity and to work toward a framework that the GRE Board Research Committee wi find usefu in assigning priorities and initiating projects. Toward that end, subsequent sections of the paper provide background, define the scope of the probem, and outine six major objectives that might guide the Board s efforts in this important area of research.

8 BACKGROUND OF THE PROBLEM To appreciate the various aspects of vaidity that appy to the GRE program, it is usefu to consider the functions and roe of the program, the kinds of research on vaidity undertaken to date, and why research on vaidity is especiay important at this point in the ife of the program. Nature of the G RE program The meaning one attaches to vaidity, as regards the GRE program, depends on how one perceives the roe of the program. The GRE program is identified primariy as a series of examinations for use by graduate schoos in seecting their students. The program has, however, a variety of functions and a variety of constituents. The foowing functions can be identified: to hep with the admission of students to graduate schoo through pubications, research, advisory services, and forum activities; to provide examination programs as measures of students potentia for success in graduate education; to provide an objective nationa basis for understanding the nature and distribution of academic taent by anayzing and describing characteristics of reevant student groups; to faciitate educationa and career guidance by providing information to students and facuty; to inform undergraduate institutions what the graduate community considers adequate preparation for graduate study. The foowing constituencies can be distinguished: students of different age, sex, background, and so forth; administrators and their institutions; different academic discipines and fieds; masters as we as doctora programs. There is considerabe variation in the extent to which the GRE program serves these functions and constituencies, and there is, no doubt, considerabe difference of opinion regarding their priority. Some have a recognized and expicit roe in the program; the roe of others may be argey impicit. For exampe, as the program has operated thus far, the roes of the ast two functions isted above are more accuratey characterized as potentiay important than as of primary concern. Aso, it seems ikey that students comprise a more important constituency of the program in their own minds than in the minds of many institutiona sponsors. Nonetheess, each of these functions and constituencies heps to define the responsibiities of the program. 2

9 In discussing the program s functions, it is necessary to distinguish between the GRE program and the examination program. The GRE program denotes the entire program structure-its organization, governance, staffing, financing, research activities, major operationa components, and so on. The examination program has a narrower meaning: the test or group of tests that a candidate may take or an institution require and the directy reated services such as directions to examinees and information for interpreting their scores, guideines for institutiona use of the tests, anaytic reports, and so forth. Uness stated otherwise, program is used hereafter to denote this narrower meaning. Previous research Research directy reated to the vaidity of the examination program has faen into one of two partiay overapping categories: (A) a variety of research carried out at ETS under GRE sponsorship and (B) vaidity studies carried out at other institutions. The overview which foows indicates the genera types of GRE research that have a bearing on vaidity and the main concusions concerning predictive vaidity that seem warranted on the basis of institutiona studies. For reviews of methodoogica or other issues, see the reports cited beow. The reevant GRE research over the past 10 to 15 years has been party concerned with traditiona vaidity studies, but aso it has treated a variety of other topics bearing upon vaidity. Part A of the attached bibiography ists 25 GRE pubications that report pertinent research. They concern the foowing topics: methodoogica issues in the conduct of vaidity studies (Bodt, 1975; Reiy and Jackson, 1974; Rock, 1974, 1975) studies of the vaidity of the GRE in the seection of foreign graduate students (Harvey and Pitcher, 1963; Sharon, 1974) test bias and the use of the GRE in seecting minority appicants for graduate study (Echternacht, 1974; Faugher, 1974) criterion probems and the anaysis of what constitutes success in graduate study (Campbe, Freund, and Lannhom, 1965; Carson, Evans, and Kuykenda, 1974; Reiy, 1974a, ) studies concerning test use in seective admissions (Bums, 1970; Campbe, Hiton, and Pitcher, 1967; Bums, Dremuk, and others, 1971; Lannhom, 1962, 1968a; Madaus, 1966) specia prediction studies and summaries of institutiona vaidity studies (Lannhom, 1960, , 1972; Lannhom, Marco, and Schrader, 1968; Lannhom and Schrader, 1951; Osen, 1955; Rock, 1972) The institutiona studies report on statistica anayses of the reationship between GRE tests and other predictors to various criteria of success in 3

10 graduate study. Wiingham (1974) has provided the most recent anaysis of a pubications and reports of vaidity studies invoving the GRE. Part B of the attached bibiography ists the 43 studies reported between 1952 and 1972 that Wiingham anayzed in that artice. These studies were based on 138 independent sets of data and 616 vaidity coefficients. The data indicated that: Vaidity coefficients for various predictors of graduate grade-point average (GPA) tend to be somewhat ower than corresponding coefficients at the undergraduate eve. This is not surprising considering the restricted range of taent frequenty encountered at the graduate eve (see, for exampe, Dawes, 1975). The undergraduate GPA is a moderatey good predictor of graduate GPA and facuty ratings; it is a poor predictor of whether a student wi attain the Ph.D. Depending upon the success criterion used, the GRE composite of Verba and Quantitative Abiity scores is either sighty or substantiay more vaid than the undergraduate GPA. The GRE Advanced Test is the most generay vaid predictor among those reviewed. It was typicay more vaid than the GRE Aptitude Test and had a higher vaidity than the undergraduate GPA in eight of the nine academic fieds represented in the review. Recommendations are a fairy poor predictor of whether a student wi successfuy compete a doctora program. A weighted composite incuding undergraduate GPA and one or more GRE scores typicay provided a vaidity coefficient in the.40 to.45 range. This was somewhat higher than the vaidity of GRE scores aone and substantiay higher than the vaidity of undergraduate GPA aone. This was the case for each success criterion and practicay every academic discipine represented. In addition to these empirica resuts, a variety of methodoogica and conceptua probems were cited that tend to create unusua difficuty in demonstrating the vaidity of entrance examinations at the graduate eve. On the basis of these probems, the data avaiabe, and other considerations, Wiingham concuded that (1) the efficiency of prediction is not ikey to be enhanced merey through the deveopment of improved predictors, and (2) the main hope for improved effectiveness in predicting success in graduate education ies in better definitions of what constitutes success, i.e., more reiabe criteria that are more ceary differentiated with respect to training objectives. The current importance of research on vaidity Considering the number of graduate programs in the country and the far reaching importance of their admissions poicies and procedures, few studies 4

11 have been made of the vaidity of the GRE for seecting graduate students. That fact and the fact that vaidity shoud aways be a prime responsibiity in any testing program are sufficient reasons for emphasizing research in this area. Furthermore, severa current circumstances make vaidity a specia concern of the GRE program. These circumstances foow three genera themes. First, the seection of graduate students is of greater concern now than in the past for the simpe reason that many more students are invoved. Seection often cannot be handed on a persona basis and, at the same time, the process is fragmented (typicay aong departmenta ines) so that the statistica technoogy of seection frequenty cannot be appied effectivey. Concurrenty, other trends are causing facuties to question the adequacy of seection practices. Undergraduate grades are assumed to be infated and ess trustworthy than in the past. New reguations to protect individua privacy give further reason to doubt the usefuness of persona recommendations. These deveopments suggest to some that GRE Verba and Quantitative Abiity scores shoud perhaps have greater weight in seection. At the same time there is increasing interest in the assessment of competence as opposed to aptitude. For exampe, it is now argued by some that seection in higher education shoud pace more emphasis on traits that come coser to the rea requirements of professiona work (Hodgkinson, 1975). In support of this view, the modest reationship between coege grades and adut success is frequenty cited (Hoyt, 1965). A these deveopments and considerations contribute uncertainty as to what constitutes a vaid basis for seecting students. Second, these educationa and methodoogica concerns are confounded by socia and ega issues that have gained great importance in the ast few years. To a considerabe extent there is de facto acceptance of an egaitarian phiosophy of admission in many institutions at the undergraduateeve-at east in the pubic sector. In arge part, admission to graduate study is sti based upon merit, but this genera rue is sharpy conditioned by the widey perceived necessity to represent fairy those groups that constitute minorities in graduate education. This necessity raises compex questions concerning what constitutes unbiased seection when prediction is, as aways, imperfect. The socia issue becomes an important ega issue when the courts are asked to decide what constitutes a vaid test and whether an institution must aways seect the student with the highest probabiity of success. Ironicay, a decision either way is ikey to raise questions of impementation that wi require far greater sophistication concerning the vaidity of admission practices than presenty exists. Whatever the resoution, when admission to priviege is treated as a ega issue, those responsibe for the process must be abe to defend its equity. The third reason vaidity is currenty such an important issue for the GRE program is that the Board is sponsoring a systematic research and deveopment effort toward program renewa, i.e., shortening the Aptitude Test, deveoping additiona modues for optiona use, and examining ways 5

12 to make the program more usefu to students and to institutions. Each of these efforts wi require carefu attention to the vaidity of proposed program changes. Not ony must presenty vaid tests and procedures be maintained; the soundness of any new conceptions regarding vaid measures and procedures in graduate admission must be demonstrated. 6

13 THE SCOPE AND MEANING OF VALIDITY OF THE GRE Often the term vaidity is conceived narrowy-simpy as a reationship between a test score and some measure of success in a subsequent activity. In considering what sorts of research on vaidity the GRE Board might want to undertake, it is necessary to take into account not ony severa conventiona conceptions of vaidity, but aso the fact that the program has various parts and various socia impications. Conventiona interpretations of vaidity The most common forms of vaidity are generay referred to as content vaidity, criterion-reated vaidity, and construct vaidity. The definitions quoted in the foowing paragraphs are taken from Standards for Educationa and Ps ychoogicu Z es ts (American Psychoogica Association, 1974). The emphasis beow has been added. Evidence of content vaidity is required when the test user wishes to estimate how an individua performs in the universe of situations the test is intended to represent. Content vaidity is most commony evauated for tests of ski or knowedge; it may aso be appropriate to inquire into the content vaidity of personaity inventories, behavior checkists, or measures of various aptitudes. Thus, content vaidity has specia reevance to the Advanced Tests since these examinations must represent subject fieds accuratey and produce appraisas of knowedge that are fair regardess of the fact that undergraduate curricuums vary from institution to institution. Criterion-reated vaidities appy when one wishes to infer from a test score an individua s most probabe standing on some other variabe caed a criterion. Statements of predictive vaidity [for exampe] indicate the extent to which an individua s future eve on the criterion can be predicted from a knowedge of prior test performance.... For many test uses, such as for seection decisions,... predictive vaidity provides the appropriate mode for evauating the use of a test or test battery. Predictive vaidity is centra to the GRE program not ony because the examinations are used to seect students ikey to succeed in graduate study, but aso because there is increasing socia and ega pressure against using tests for such purposes uness there is cear pubic evidence of such a reationship. Evidence of construct vaidity is not found in a singe study; rather, judgments of construct vaidity are based upon an accumuation of research resuts. In obtaining the information needed to estabish construct vaidity, the investigator begins by formuating hypotheses about the characteristics of those who have high scores on the test in contrast to those who have ow scores. Taken together, such hypotheses form at east a tentative theory 7

14 about the nature of the construct the test is beieved to be measuring. In considerabe part, the construct vaidity of the GRE rests upon decades of psychometric research, indicating that verba and quantitative abiity pay a critica roe in most types of inteectua work, and upon even more extensive educationa experience which indicates that frequenty the best predictor of future success in an academic fied is eary competence indicated by a subject-matter test. Construct vaidation requires constant attention, however, to insure that a test is actuay measuring the construct intended. For exampe, it is necessary to insure that a reading comprehension test is not so compicated in content as to stress reasoning instead of reading, or that a mathematics test does not use anguage that paces a premium upon knowedge of vocabuary. Naturay, construct vaidation is even more demanding and important in the case of new measures in areas ike cognitive stye and creativity. Socia interpretations of vaidity A number of broad interpretations of vaidity are associated with the socia impications of test use. For exampe, there are such questions as thevaidity of a test for different groups of peope, test vaidity as refected in the ways test use affects the users, and onger range effects of using a particuar test in a arger socia context. These are more recent interpretations of vaidity. They deserve specia consideration because the GRE program operates in an unusuay broad socia context. First, a vaid test must be fair and appropriate for a individuas taking the test. That is, it must be free of systematic bias and distortion vis-&is the various popuations or subgroups taking the test, and the test shoud not have different meanings for such groups. Messick (1975) makes the important point that one vaidates not a test, but an interpretation of data derived from a specific procedure. Whether that procedure (test) has the same properties and patterns of reationships in different popuation groups is an important empirica question. From a somewhat different ange, Thorndike and Hagen (1969) refer generay to vaidity as whether the test measures what we want it to measure. Thus, whie a test may be intended to measure knowedge of American history, there are a number of things it is not intended to measure; e.g., reading speed, cutura disadvantage, sex, age, anguage spoken in the home, and so on. In this sense there are an indefinite number of ways in which a test may be biased and an indefinite number of subgroups for which a test may not be appropriate. There is no way to guard against a such possibiities, and it can easiy happen that making a test fairer for one person may make it ess fair for another. It is evident, however, that bias and ap-

15 propriateness are important aspects of vaidity that require constant attention. A second socia interpretation of vaidity concerns the use and usefuness of a test in the context in which it is actuay appied. The casua phrase test vaidation seems to impy that the score one interprets comes from a naked instrument. The instrument however is ony one eement in a procedure and a vaidation study examines a procedure as a whoe (Cronbach, 1971). Tests are constructed with a purpose in mind and they are used in a context of instructions, interpretative materias, supporting research, expected effects, and constanty changing conditions of use. Again, vaidity does not reside in the test itsef, but depends upon appropriate outcomes resuting from the use of the test. The GRE Board cannot be responsibe for every conceivabe instance of test misuse, but it shoud assume responsibiity for making known the conditions of proper use, for advancing understanding of the socia and educationa impications of different uses of tests, and for insuring that tests are not presented in ways that might encourage inappropriate use. There is another important reationship between vaidity and test use. It is especiay true in a context ike the GRE program that a test or measure does not stand aone. It competes for time and attention with other parts of the examination program that may be equay vaid for the same purpose or more vaid for other purposes. This eads to a third socia interpretation of vaidity. Cronbach (1971) distinguishes educationa importance as a form of vaidity equa in stature, and parae, to content vaidity and construct vaidity. He defines this form of vaidity as foows: Does the test measure an important educationa outcome? Does the battery of measures negect to observe an important outcome. 7 The choice and content of measures incuded in the Board s examination program are significant because those measures constitute an important socia communication. Regardess of whether they are intended as such, the GRE do to some extent communicate to coeges what the graduate community thinks coeges shoud teach and what students shoud earn. It is aso argued that the GRE program needs to refect the important earning outcomes of undergraduate education; i.e., it must foow the curricuum instead of eading it. From either point of view, the content of the program constitutes a message that has a bearing on education far beyond the admissions process. Consequenty the inherent reevance, significance, and vaue of the traits measured deserve cose attention. These atter interpretations of vaidity bear especiay upon the usefuness and appropriateness of different components of the examination program, both as they serve the immediate purposes for which they are intended and as they may be justified in some broader educationa sense. Obviousy such interpretations do not appy to isoated measures, but to the program as a whoe. These considerations and the foregoing discussion suggest a broader notion-one of program vaidity as distinct from test vaidity. 9

16 Construct vaidity of the program Principes concerning vaidity can be appied to tests and other individua measures, to batteries or groups of measures, or to an integrated program that may have a variety of pieces (e.g., a centra core, a variety of test options, biographica information, specia measures, and so on). Each of those pieces shoud have a rationae concerning its egitimate and usefu function. But the pieces are not free-standing. The various components are used in a context of reated procedures, materias, services, and so on. Each part of the program is to some extent dependent upon other parts and upon an overa rationae as to how the program serves its functions and its constituents. These considerations suggest that the Board shoud be guided by an overarching sense of the construct vaidity of the program. In this context, a vaid test is a defensibe test; i.e., an accurate and fair measure of what you want to measure and aso one that is usefu for its purpose. More specificay, the notion of the construct vaidity of a program suggests that a test or measure is a vaid component of a program if it meets these conditions: It represents fairy what is intended. That is, it satisfies concerns such as content vaidity, construct vaidity, educationa importance, and appropriateness for the examinees, both as one group and as subgroups. Its use is demonstraby effective. It meets the requirements of criterion reated vaidity, predictive bias, characteristics of the program that affect test use, and ega issues concerning test use. It serves a distinctive purpose in reation to other tests and measures in the program, that is a purpose not served by the other tests and measures. 10

17 The previous discussion provides background for the probem. In this section six research objectives are suggested in order to provide for the GRE Board Research Committee s consideration specific proposas for action. The objectives are as foows: I. To encourage and faciitate institutiona vaidity studies II. To dea effectivey with methodoogica issues concerning vaidity that require the GRE program s initiative II I. To deveop improved criteria of success in graduate study IV. Popuation vaidity: How to improve it and enhance understanding of it V. To improve institutiona use of summary program data VI. To systematicay insure the vaidity of revised or new measures resuting from program renewa In the foowing paragraphs an initia statement of each objective is foowed by a brief rationae and discussion of severa issues reevant to the objective. These issues are discussed either as genera research needs or, in some cases, as more specific possibe projects. But the main purpose is to suggest a framework for thinking about vaidity research that is needed. Objective I: To encourage and faciitate institutiona vaidity studies The American Psychoogica Association (1974) outines a variety of responsibiities of test sponsors for examining and estabishing the vaidity of measures they offer for use. In this paper we give specia attention to these responsibiities of the GRE program, but the conditions under which tests may be vaid or invaid are essentiay unimited because appications vary so widey with respect to purpose, academic fied, criteria, oca conditions, and so on. The GRE Board cannot hope to estabish vaidity in even a significant minority of the possibe situations in which the tests may be used. Consequenty, it is important for users to recognize their own responsibiity for examining the vaidity of a test for the purpose and circumstances they have in mind. As Cronbach (1971) states, In the end, the responsibiity for vaid use of a test rests on the person who interprets it. The pubished research merey provides the interpreter with some facts and concepts. He has to combine these with his other knowedge about the persons he tests and the assignments or adjustment probems that confront them, to decide what interpretations are warranted. But users confront many probems in carrying out institutiona vaidity studies. In most cases the appropriate ocae is the individua department where, however, the number of students may be sma and the facuty may ack sufficient interest or expertise to 11

18 pursue the question of vaidity. It is important, therefore, that the GRE program find ways to encourage and faciitate institutiona vaidity studies. One possibiity woud be to deveop a imited program of cooperative vaidity studies. This might invove identification of individua departments or institutions where there is the interest and possibiity of carrying out a study of more than routine interest. With technica advice from staff at Educationa Testing Service (ETS), the institution might organize and suppy appropriate data; a researcher in Princeton might anayze the data and prepare a report, probaby in some mode format. This process might aso invove identification and announcement of priority areas of interest; e.g., particuar academic fieds, interesting possibiities for criterion deveopment, specia popuations of students, and institutionwide use of program information. Periodicay such group studies might be coected and reported. Another possibe approach woud be to deveop a vaidity study kit that might consist of a step-by-step notebook for doing oca studies, usefu references and forms, a coection of reevant reprints, and so on. This mode for encouraging oca studies has the virtue of cost-effectiveness, but it aso paces most of the responsibiities for initiative on the institution. ETS s main responsibiity woud be to produce the kit. That itsef wi require experience and good ideas if such a kit is to be usefu and cope effectivey with the variation of oca circumstances. Another research need somewhat reated to those above is the desirabiity of deveoping effective reationships with institutions in order to faciitate work in this area. On the one hand, both of the above possibiities can be greaty faciitated by working intensivey with one or two institutions over a reasonabe period of time to expore the probems of conducting institutiona vaidity studies at the graduate eve. Furthermore, there wi ikey be a need to deveop a cooperative reationship with a variety of institutions in order to vaidate experimenta modues that may be considered for incusion in the GRE program over the next severa years. This needis directy reated to Objective VI, though the deveopment of the necessary institutiona reationships wi profit from groundwork prior to the time when the need actuay arises. Objective II: To dea effectivey with methodoogica issues concerning vaidity that require the GRE program s Initiative A variety of traditiona issues typicay referred to as technica probems make vaidity studies exceptionay difficut at the postgraduate eve. Perhaps the most serious is the criterion probem, though it is a transcendent issue concerning conception as we as methodoogy and deserves separate 12

19 discussion as Objective III beow. The most famiiar technica issues concern very sma sampes, often severe restriction in the range of taent due to seection, and ack of confidence in the meaning and reiabiity of undergraduate grades. The foowing paragraphs outine some of these issues and suggest some possiby fruitfu ines of research. Vaidity studies carried out in individua departments are often based upon sma sampes and a very restricted range of test scores and undergraduate grades. These conditions often combine to produce ow and erratic vaidity coefficients. A reated but different view of this probem is the fact that there has been very itte attention given to the vaidity of the GRE among departments within individua discipines or fieds. It woud appear desirabe to give additiona attention to ways of pooing data across departments, thereby mitigating the technica probems and aso demonstrating vaidity in a arger context. This requires use of some common criterion, Perhaps the ony one that woud make sense is some genera notion of success in graduate education, such as competion of the degree or overa facuty ratings. Another genera possibiity for deaing with these issues is a retrospective nomination study; i.e., asking facuty in a number of departments within a fied to nominate outstanding and poor students over a period of severa years. It may be possibe to deveop substantia sampes for studies within seected fieds. This type of study woud aso require a common criterion of success, such as obtaining the Ph.D. A specia advantage of these two types of studies is the fact that they can be carried out over a imited time period. But, in addition to narrowy conceived vaidity studies, many especiay interesting research questions require ongitudina study. Serious consideration shoud be given to the deveopment of a ongitudina study that foows a carefuy structured sampe of students through and beyond graduate education. Students from severa fieds might be incuded with oversamping of specia groups of interest. With severa spaced foow-ups, a group of cooperating students coud provide a vauabe data base for a variety of studies in addition to specific investigations designed at the outset. Topics of specia interest in such a ongitudina study woud incude foow-up of minority students through and beyond graduate education, studies of patterns of attendance and career choice, and anaysis of the cost of graduate education and how financing aternatives affect students decisions. The quaity of undergraduate grades as a predictor is another famiiar probem. It is commony understood and accepted that a B at one institution is not necessariy equivaent to a B at another institution. A good dea of research at the undergraduate eve_$as indicated that there is no vaue in trying to adjust grades from different high schoos if admissions decisions are made on the basis of grades and-an entrance examination. At the graduate eve, however, it seems that corrections for variations in grading standards from one undergraduate coege to another can sometimes improve predictions sighty. Pitcher and Schrader (1972) report that mutipe 13

20 correations predicting success in graduate business schoos are increased on the average about.02 when the quaity of the undergraduate institution is taken into account. A simiar resut has been found in the case of aw schoo admissions. This matter may be worth an exporatory investigation in connection with the GRE, though it is not cear whether the potentia gain woud make such an investigation worthwhie, nor whether it woud be easy to impement the resuts in any event. Furthermore, appying institutiona corrections may invove difficut poitica probems in graduate admissions. A more serious question may be the status of undergraduate grades in genera. Informa opinions suggest a mounting ack of confidence in the reiabiity and dependabiity of undergraduate grades as a predictor due to the fact that grades are severey infated and the fact that some facuty are opposed to competitive grading as a matter of principe. It is difficut to judge the extent to which undergraduate grades have actuay been seriousy compromised as a measure of accompishment and predictor of subsequent academic success. A carefu study of coege grading, perhaps with resuting recommendations from the Board, coud possiby be of great vaue in reversing an undesirabe trend or, at the very east, in reveaing the character and scope of the probem and suggesting ways of deaing with it. A reated issue is the probem of interpreting the credentias of undergraduate students when they do not come in the form of grades at a An increasing number of institutions are recognizing various forms of nontraditiona earning, converting to competency-based curricuums, awarding credit for experientia earning, and experimenting with narrative transcripts. A study of how such credentias are evauated and to what extent they forecast success in graduate study coud provide a vauabe service by heping to infuence evoving nontraditiona practices in sound directions. Objective III: To deveop improved criteria of success in graduate study It is commony acknowedged that a principa difficuty in estabishing the vaidity of predictors and procedures for admitting graduate students is the ack of cear-cut reiabe criteria of success. Grades are widey suspect; methods of evauating students performance in graduate education vary greaty from department to department; and the utimate criterion of degree attainment takes many years to ascertain and depends upon many intangibe and fortuitous events. To a considerabe extent the criterion probem is due to the very common tendency of those responsibe for educationa seection programs to focus upon predictors and to ignore the measures of achievement that those predictors are intended to forecast. The need to give specific attention to the rationae and measurement (i.e., construct vaidity) of criteria is we-stated by Cronbach (1971): The 14

21 r asymmetric conception of the test as a predictor of a certain performance has been discarded in favor of a symmetric view. According to this, persons are observed in situations. Some are artificia occasions for observations, which are caed tests, and some are situations arising in the natura course of the person s work or schooing. Reating these observations to each other tes one about the situationa demands and about the resources individuas bring to bear. To study the vaidity of a test interpretation is to study how behavior in one situation is reated to behavior in another. Both observations revea characteristics of,the individua, and both types of behavior shoud be understood. Defensibe and reiabe criteria of success are ikey to become more important in the face of agging confidence in grades and the increasing need to justify administrative actions, both with respect to admitting and dropping students. Furthermore, admission standards are ikey to come under increasing scrutiny, party by those speaking for underrepresented groups who question the socia equity of current practices and party by those who assert there is undue reiance upon aptitude tests and objective measures in genera. Shoud ega action require empirica justification of admission decisions, the need to deveop sound criteria wi immediatey become critica. Wiingham (1974) has urged much greater attention to the probem of criteria, especiay in the context of a broader view of predictor-criterion reationships and aternate strategies of seection. The aternate strategies depicted in Figure 1 impy that different departments or programs within departments may emphasize different training objectives, which in turn shoud be reated to the way students are seected and the way their performance is evauated. PREDICTORS specia accompishments. undergraduate grades abtty tests achievement tests creatwty cogntve styes background characterstcs PROGRAM OBJECTIVE INTERMEDIATE ON-THE-JOB (TO TRAIN THE) CRITERIA CRITERIA PRACTITIONER - F TEACHER e demonstrated sk and Interest I practica probems eary invovement m professiona afaw Intern performance academic competency teaching skis demonstrated interest and sku m heppg students Invovement m nsttutona affairs academic competency I r 9 Independent r professtona eadershp student and aum171 judgment of coeagues facuty eadership pubcatons accomphshments ctatons SCHOLAR/SCIENTIST - novat ve work ) awards pub\cat!ons eminence academic competency nventons Figure 1: L Aternate prediction strategies in graduate education. Reproduced with permission from Science, 1974, 183, (4122), 277. Copyright 1974 by the American Association for the Advancement of Science. 1 15

22 Another aspect of Figure 1 deserves specia attention. Moving from eft to right in the figure, predictors, intermediate criteria, and on- the-job criteria form a prediction chain. Preadmission predictors such as test scores or specia achievements are intended to predict success in graduate schoo (an intermediate criterion). Performance in graduate schoo shoud, in turn, predict success in subsequent career-reated activity. Since utimate career success depends upon many circumstances and deveoped skis not necessariy reated to performance in graduate schoo or to earier predictors, it is improper to think of a graduate admission test as a predictor of job success. On the other hand, if a test or other measure is used to screen students, it ought to be hepfu in predicting success in schoo and aso shoud have a ogica reationship to ong-range criteria. That is, the test shoud have construct vaidity in the sense that it represents an abiity of demonstrabe importance (perhaps one of many) in determining ong-range success. Deveoping a better conception of intermediate criteria of success in graduate schoo is an important step in examining the ogica reationships in the prediction chain. There are severa strategies that might be hepfu in deveoping improved criteria. It seems especiay important to encourage systematicay the deveopment of better criteria in the routine execution of vaidity studies as we as the routine evauation of students performance. This might be accompished party by working with individua institutions to carry out and pubish mode studies that can hep to iustrate the deveopment of different types of criteria. For exampe, it woud be desirabe to iustrate and encourage the use of reiabe criteria ike rating scaes, such as those deveoped through GRE research (Carson, Reiy, Mahoney, and Cassery, 1976), or comprehensive examinations, which may aready be avaiabe within departments but are not normay used in vaidity studies. Considering the importance of criteria and the critica roe they coud pay given additiona ega interest in graduate admissions, it may be desirabe to undertake a fairy systematic anaysis of how graduate departments view success. A usefu foow-up to the Carson, Evans, and Kuykenda (1974) survey woud be an intensive anaysis of the rationae and basis upon which departments evauate their students, the evauation procedures actuay empoyed, and the psychometric properties of the resuting criteria. This sort of anaysis coud have considerabe vaue in describing how high eve taent is currenty assessed and in suggesting opportunities and possibe road bocks in the deveopment of improved criteria. Another generay desirabe strategy woud be to foster the deveopment of such intermediate criteria as depicted in Figure 1. This might invove facuty ratings of particuar types of accompishments, specia means of coecting outside judgments, or whatever procedures may be required to obtain information that is reevant to the most important training objectives. One way of encouraging the deveopment of such criteria is through the cooperative institutiona studies described under Objective I. Another approach is a specia deveopmenta project which may be required in the 16

23 case of an unusuay compex criterion such as scientific creativity. The present GRE research on scientific creativity is concerned specificay with the deveopment of an intermediate criterion. Another potentiay usefu approach starts with the observation that practicay a vaidity studies incorporate criteria based upon student performance in graduate schoo. This typica design has two shortcomings if one is interested in confirming the utimate socia reevance (i.e., construct vaidity) of the GRE. First, typica vaidity studies are not directy reevant to the question of whether the GRE are effective in seecting for graduate study peope who are ikey to reach the highest eves of professiona success. Whie screening prospective professionas from a group of graduate students is primariy the responsibiity of the graduate schoos, tests used for earier screening shoud not be counterproductive in that process; that is, one woud ike to know that very successfu professionas have typicay scored we so that screening out students with ow scores is both efficient and defensibe. Second, studies that use success criteria reative to the standards of individua institutions may seriousy underestimate the usefuness of the examinations because the range of taent is typicay restricted at individua institutions, but ranges widey from one institution to another. Thus, the GRE may be reativey poor predictors of graduate performance in a singe prestigious history department, but may provide a reasonaby good indication of differentia competence among graduates of a history departments. It might be worthwhie, therefore, to examine the feasibiity of determining Aptitude Test score eves for pertinent groups of individuas who have achieved some forma measure of success in their fied. These might incude such ad hoc groups as feows of earned societies, officers of professiona organizations, facuty of prestigious departments, individuas isted in honorific biographies, and so on. Comparison of the scores of such individuas with appropriate normative groups woud be interesting, even admitting the possibiity that some individuas may achieve prestigious status party because at one time they were known to have scored highy on the test, or in spite of having scored poory. Objective IV: Popuation vaidity: How to improve it and enhance understanding of it Messick and Barrows (1972) used the term popuation vaidity in referring to the generaizabiity of research findings across different popuations. A simiar notion appies to the vaidity of tests and other psychometric measures. If a test eads to incorrect inferences about a particuar popuation, then the test is to that extent invaid. Incorrect inferences may resut from the fact that the test itsef is not a good measure for that popuation or that the test does not have the same reationship with the criterion for that popu- 17

24 ation. We can refer to these two sources of error as appropriateness and predictive bias. Research on test appropriateness has been imited argey to item-group interaction studies and has not been especiay productive as yet. More attention has been directed to predictive bias and most such studies have compared predicted and actua performance of back and white students. Linn s (1973) review of that research suggests a sma but fairy consistent tendency for tests to overpredict the coege grades of back students. Recenty, attention has been directed to technica factors such as reiabiity that might expain that finding (Linn, Note 1). A significant recent deveopment has been the systematic anaysis of the psychometric characteristics and socia impications of different seection modes (Petersen and Novick, 1974). Somewhat to the surprise of many peope working in this fied, it is now apparent that there are different seection approaches with rather different impications that are compex both technicay and poiticay. For exampe, it can be demonstrated that a seection mode that is fair to an individua may not be fair to a socia group to which that individua beongs. These different seection modes may give rise to ega compications and, if so, perhaps even greater compications for the GRE program and for admission committees. The resoution of these issues may be argey poitica, but it seems important for the Board to pursue reevant research and other activities in this area that wi hep to iuminate the issues and carify the impications for the program. Severa possibiities can be suggested. The program has undertaken severa studies to anayze item-group interactions in an effort to identify types of items that might be unfair to some popuations of examinees. That work has not been fruitfu in discovering significant numbers of such items or in deveoping effective ways of identifying an unfair item prior to its use. More recent anaysis of these types of data revea some sma but interesting tendencies for certain types of items to be harder or easier for different subgroups of examinees. In most cases it seems doubtfu that these differences can be termed bias in any reasonabe sense of the word, but the ine of inquiry is usefu from a research standpoint and desirabe from the standpoint of monitoring the fairness of the GRE. A promising and cosey reated ine of research is the possibiity of a more genera attack on the question of the appropriateness of a test for the various popuations of peope who might take it. The Board has underway a deveopmenta project that may succeed in creating appropriateness indices that woud permit study of this probem from a different vantage point. The work is argey mathematica and, if successfu, wi ikey take severa years to improve our understanding of popuation vaidity. Whie the Board must be concerned about the possibiity of predictive bias in reation to any popuation of examinees, a specia probem exists with respect to individuas who have been out of the educationa system for an extended period of time. There is a common assumption that an objective examination is to some extent unfair to an individua of 35 who has been 18

25 away from coege for a dozen years. In considerabe part this probem paraes the issue of test bias with respect to minorities and can be studied in a simiar fashion. A substantia iterature has deveoped concerning the technica, educationa, and ega issues surrounding the genera matter of popuation vaidity. There are reviews of various aspects of test bias, ega decisions, and impications for admissions committees. These probems are of considerabe concern to departmenta facuties, though the iterature is ikey not we known or avaiabe to them. The Board might consider whether the pubication of a imited group of reprints with commentary woud be ausefu service beyond its present activities in this area. Objective V: To improve institutiona use of summary program data Vaidity is often conceived of ony in reation to the individuas who take a test. But test performance data are often reported about groups of examinees and inferences are drawn concerning those groups and the educationa programs in which they have taken part. As the GRE program seeks to serve better the interests and needs of institutiona sponsors, such summary data shoud be reported more frequenty and systematicay. The prospect of more systematic reporting of program data intensifies the need to insure that such data serve the usefu purposes intended and do not foster erroneous inferences. Two exampes iustrate the opportunity and the need. The GRE program generates biographica and test data that can hep institutions to understand the dynamics of the admissions process and possiby ater the process to serve their institutiona purposes. The fow of taent into and through graduate education can be conceived of as a diminishing poo of individuas; viz., the poo of appicants, the group offered admission, the group who actuay enro, and finay the group who obtain a degree. Data from the program can be used not ony to describe the characteristics of these successivey diminishing groups, but aso to identify the persona and demographic characteristics that operate most strongy in the seection process preceding each stage. Such anaysis shoud hep educationa panners make usefu connections between resource aocations, program panning, and encouragement of taent. As a second iustration: the Board has taken steps to reate the GRE program to the Undergraduate Assessment Program at ETS. One aspect of this new service is to deveop additiona subscores for the GRE Advanced Tests and report them in summary form for groups of students in undergraduate departments. This sort of reporting can be quite usefu to departments if it emphasizes the comparison of departmenta objectives with student performance on corresponding parts of the examinations. The design of an interpretive framework of this sort coud be an important contribution of 19

26 the GRE Board in improving cooperation between undergraduate and graduate education. Objective VI: To systematicay insure the vaidity of revised or new measures resuting from program renewa In recent meetings the GRE committees and the GRE Board have given carefu consideration to a broad effort aimed at program renewa over the next severa years. Naturay this effort is experimenta and highy tentative, but it does incude a number of possibe changes or additions to the GRE program-among them a possibe shortening of the Aptitude Test and the deveopment of optiona modues that might be incorporated in the test, improvements in the Advanced Tests, instruments that might be usefu for purposes other than seective admissions, and improved anciary services for institutions and students. It is impossibe to know which of these promising possibiities wi eventuate, but the process of program renewa paces a specia responsibiity on the Board to insure the continuing integrity and vaidity of the program. There is firm consensus that renewa is important and that the vaidity of the program wi thereby be enhanced, but a compex nationa program has ripping effects and reationships that are often difficut to anticipate. Consequenty, change shoud not be taken ighty. It shoud be taken ony on the basis of reasonabe assurance that revised or new components have demonstrabe vaidity as they were intended. Such assurance shoud be a conscious objective of the Board. Specific necessities for vaidation wi arise as new program components are deveoped. Possibiities aready apparent incude the foowing: examination of the factoria and predictive vaidity of new versus od forms of the verba and quantitative Aptitude Test; deveopment of a normative framework for the interpretation and possibe operationa use of a measure of cognitive stye; construct and criterion-reated vaidation of objectivey scored intermediate criteria of scientific creativity; content and predictive vaidation of an inventory of documented accompishments. These six objectives do not exhaust the range of important priorities concerning vaidity, nor are the possibiities mentioned under each objective intended to be incusive. They are, however, indicative of the very diverse issues that require attention if the GRE Board is to fee reasonaby confident that its examination program does exhibit construct vaidity in a essentia respects. In that spirit the foregoing shoud be viewed as a possiby usefu framework and orientation, not a prescription. 20

27 STATUS OF RESEARCH RELEVANT TO THE SIX OBJECTIVES The foowing outine ists under each objective the reevant current GRE research as we as projects conducted and reported upon in the past. As suggested by the previous discussion, however, there is need for further research of various types. Some has aready been proposed to the GRE Board Research Committee, some is sti in the ream of possibiity. Objective I: To encourage and faciitate institutiona vaidity studies Reports Predicting graduate schoo success (Lannhohn and Schrader, 1951) Predicting success in Yae Schoo of Forestry (Osen, 1955) Abstracts of seected vaidity studies (Lannhom, 1966) Review of vaidity studies (Lannhom, 1968a) Cooperative vaidity studies (Lannhom, Marco, Schrader, 1966) Summary of vaidity studies (Lannhom, 1972) Predicting success in graduate education (Wiingham, 1974) Omen t Project Cooperative institutiona vaidity studies (Wison) Objective II: To dea effectivey with methodoogica issues concerning vaidity that require the GRE program s initiative Reports The test chooser (Rock, 1974) Effects of option weighting on vaidity (Reiy and Jackson, 1974) Popuation moderators (Rock, 1975) Bayesian and east squares prediction (Bodt, 1975) Prediction of Ph.D. attainment in psychoogy, mathematics, and chemistry (Rock, 1972) Objective III: To deveop improved criteria of success in graduate study Reports A study of the Advanced History Test (Campbe, Freund, and Lannhom, 1965) Critica incidents of graduate performance (Reiy, 1974a) Factors in graduate performance (Reiy, 1974b) 21

28 Feasibiity of common criterion vaidity studies (Carson, Evans, and Kuykenda, 1974) Criterion rating scaes (Carson, Reiy, Mahoney, and Cassery, 1976) Current Projects Intermediate criteria of creativity (Frederiksen and Ward) How graduate departments assess students (Wison) Objective IV: Popuation vaidity: How to improve it and enhance understanding of it Reports Prediction of graduate performance of foreign students (Harvey and Pitcher, 1963) Use of TOEFL and GRE in predicting success of foreign students (Sharon, 1974) A quick method of determining test bias (Echternacht, 1974) New definitions of test bias (Faugher, 1974) Current Projects Deveopment of an appropriateness index (Levine) Guessing instructions and subgroup performance (Pike) Objective V: To improve institutiona use of summary program data Current Projects Deveopment of additiona psychoogy subscores ( Atman and McPeek) Objective VI: To systematicay insure the vaidity of revised or new measures resuting from program renewa Current Projects Cognitive stye ongitudina study (Witken and Ward) Machine-scored fied dependence test (Reiy and Donon) Vaidity of scientific thinking tests (Frederiksen) Inventory of accompishments (Baird) Machine-storabe tests of scientific thinking (Frederiksen and Ward) Factor anaytic study of current and proposed forms of the GRE Aptitude Test (Powers, Swinton, and Carson) Research into shortening V and Q and adding a new measure to the Aptitude Test (Conrad and Atman) 22

29 Note that GRE Board pubications with a P in the pubication number are of interest primariy to researchers. References American Psychoogica Association. Standards for educationa and psychoogica tests. Washington, D.C.: APA, Carson, Afred B., Evans, Frankin R., and Kuykenda, Nancy M. The feasibiity of common criterion vaidity studies of the GRE (GRE Board Professiona Report 71-1P). Princeton, N. J. : Educationa Testing Service, Juy 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Memorandum RM-73-16, Juy Carson, Afred B., Reiy, Richard R., Mahoney, Margaret H., and Cassery, Patricia L. The deveopment and piot testing of criterion rating scaes. (GRE Board Professiona Report 73-1P). Princeton, N.J.: Educationa Testing Service, October Cronbach, Lee J. Test vaidation, in Thorndike, Robert L. (Ed.), Educationa measurement (2nd ed.). Washington, D.C.: American Counci on Education, Dawes, Robyn M. Graduate admission variabes and future success. Science, 1975, 187 (4178): Hodgkinson, Harod. Cited in NIE s new director questions credentiaing. Education Daiy, June 24, 1975, p. 3. Hoyt, Donad P. The reationship between coege grades and adut achievement: A review of the iterature (ACT Research Report Number 7). Iowa City, Iowa: American Coege Testing Program, (ERIC Document Reproduction Service No. ED ) Linn, Robert L. Fair test use in seection. Review of Educationa Research, 1973, 43 (2): Messick, Samue. The standard probem: Meaning and vaues in measurement and evauation. American Psychoogist, 1975, 30 (10): Messick, Samue, and Barrows, Thomas S. Strategies for research and evauation in eary chidhood education, in Nationa Society for the Study of Education, Seventy-first yearbook, Part II. 1972, pp

30 Petersen, N.S., and Novick, M.R. An evauation of some modes for test bias (ACT Technica Buetin No. 23 and Iowa Testing Program Occasiona Paper Number 8). Iowa City, Iowa: American Coege Testing Program, September Pitcher, Barbara, and Schrader, Wiiam B. Indicators of coege quaity as predictors of success in graduate schoos of business (Admission Test for Graduate Study in Business Brief Number 6). Princeton, N. J. : Educationa Testing Service, Thorndike, Robert L., and Hagen, Eizabeth. Measurement and evauation in psychoogy and education (3rd ed.). New York: John Wiey and Sons, Wiingham, Warren W. Predicting success in graduate education. Science, 1974, 183 (4122): Reference note Note 1. Linn, Robert L. Test bias and the prediction of grades in aw schoo. Paper prepared for the Conference on the Future of Law Schoo Admission Counci Research, September 27-28, Bibiography-Part A Bodt, Robert F. Comparison of a Bayesian and a east squares method of educationa prediction (GRE Board Professiona Report 70-3P). Princeton, N. J.: Educationa Testing Service, June Previousy pubished as Research Buetin RB-75-15, Apri Burns, Richard L. Graduate admissions and feowship seection poicies and procedures, Part I and Part II (a report of the GRE Board). Princeton, N. J.: Educationa Testing Service, Burns, Richard L., Dremuk, Richard, Hein, Andrew J., Robey, Richard C., Scudder, Hervey C., and Witsey, Robert G. Graduate admissions and feowship seection poicies and procedures: Case studies (a report of the GRE Board). Princeton, N. J.: Educationa Testing Service, Out of print. (ETS Archives Microfiche #491) Campbe, Joe T., Freund, Lucy, and Lannhom, Gerad V. A study of performance on the Advanced History Test (GRE Specia Report 65-2). Princeton, N. J. : Educationa Testing Service, August

31 Campbe, Joe T., Hiton, Thomas L., and Pitcher, Barbara. Effects of repeating on test scores of the Graduate Record Examinations (GRE Specia Report 67-). Princeton, N.J.: Educationa Testing Service, Apri Carson, Afred B., Evans, Frankin R., and Kuykenda, Nancy M. The feasibiity of common criterion vaidity studies of the GRE (GRE Board Professiona Report 71 -P). Princeton, N. J. : Educationa Testing Service, Juy 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Memorandum No , Juy Echtemacht, Gary. A quick method for determining test bias (GRE Board Professiona Report 70-8P). Princeton, N. J. : Educationa Testing Service, Juy 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Buetin RB Apri Faugher, Ronad L. The new definitions of test fairness in seection: Deveopments and impications (GRE Board Research Report ). Princeton, N. J.: Educationa Testing Service, May 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Memorandum RM-73-17, September Harvey, Phiip R., and Pitcher, Barbara. The reationship of Graduate Record Examinations Aptitude Test scores and graduate schoo performance of foreign students at four American graduate schoos (GRE Specia Report 63-). Princeton, N.J.: Educationa Testing Service, Apri Lannhom, Gerad V. Abstracts of seected studies on the reationship between scores on the Graduate Record Examinations and graduate schoo performance (GRE Specia Report 60-3). Princeton, N. J. : Educationa Testing Service, November Out of print. (ETS Archives Microfiche R236) Lannhom, Gerad V. The use of Graduate Record Examinations in appraisinggraduate study candidates. (GRE Specia Report 62-3). Princeton, N.J.: Educationa Testing Service, October Lannhom, Gerad V. Review of studies empoying GRE scores in predicting success in graduate study, (GRE Specia Report 68-1). Princeton, N.J.: Educationa Testing Service, March 1968a. Lannhom, Gerad V. The use of GRE scores and other factors in graduate schoo admissions (GRE Specia Report 68-4). Princeton, N. J.: Educationa Testing Service, October 1968b. Lannhom, Gerad V. Summaries of GRE vaidity studies (GRE Specia Report 72-). Princeton, N. J.: Educationa Testing Service, February

32 Lannhom, Gerad V., Marco, Gary L., and Schrader, Wiiam B. Cooperative studies of predicting graduate schoo success (GRE Specia Report 68-3). Princeton, N. J. : Educationa Testing Service, August Lannhom, Gerad V., and Schrader, Wiiam B. Predicting graduate schoo success: An evauation of the effectiveness of the Graduate Record Examinations. Princeton, N.J.: Educationa Testing Service, Out of print. (ETS Archives Microfiche #4 74) Madaus, George F. The deveopment and use of expectancy tabes for the Graduate Record Examinations Aptitude Test (GRE Specia Report 66-). Princeton, N.J.: Educationa Testing Service, Apri Osen, Marjorie. The predictive effectiveness of the Aptitude Test and the Advanced Bioogy Test of the GRE in the Yae Schoo of Forestry (Statistica Report Number 55-6). Princeton, N.J.: Educationa Testing Service, Out of print. Reiy, Richard R. Critica incio?ents of graduate student performance (GRE Board Research Report 70-5). Princeton, N. J. : Educationa Testing Service, June 1974a (ERIC Document Reproduction Service No. ED ). Previousy pubished as GRE Board Technica Memorandum No. 1, Apri Reiy, Richard R. Factors in graduate student performance (GRE Board Professiona Report 71-2P). Princeton, N. J. : Educationa Testing Service, Juy (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Buetin RB-74-2, February Reiy, Richard R., and Jackson, Rex. Effects of empirica option weighting on reiabiity and vaidity of the GRE (GRE Board Professiona Report 71-9P). Princeton, N.J.: Educationa Testing Service, Juy 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Buetin RB-72-38, August Rock, Donad A. The prediction of doctorate attainment in psychoogy, mathematics, and chemistry (GRE Board Preiminary Report). Princeton, N. J.: Educationa Testing Service, August 1972 (ERIC Document Reproduction Service No. ED ). Later pubished as GRE Board Research Report 69-6aR, June Rock, Donad A. The test chooser : A different approach to a prediction weighting scheme (GRE Board Professiona Report 70-2P). Princeton, N.J.: Educationa Testing Service, November

33 Rock, Donad A. The identification of popuation moderators and their effect on the prediction of doctorate attainment (GRE Board Professiona Report 69-6bP). Princeton, N. J. : Educationa Testing Service, February Sharon, Amie T. Test of Engish as a Foreign Language as a moderator of Graduate Record Examinations scores in the prediction of foreign students grades in graduate schoo (GRE Board Professiona Report 70-1P). Princeton, N.J.: Educationa Testing Service, June 1974 (ERIC Document Reproduction Service No. ED ). Previousy pubished as Research Buetin RB-71-50, September Bibiography-Part B The vaidity studies in the bibiography beow were anayzed in Predicting Success in Graduate Education, an artice written by the author of this paper and pubished in Science, 1974, Vo. 183, No The studies concern the period About haf were pubished, and haf were institutiona reports or theses. Aexakos, Constantine E. The Graduate Record Examinations: Aptitude Tests as screening devices for students in the Coege of Human Resources and Education. Unpubished report, West Virginia University, Reported by G.V. Lannhohn in GRE Specia Report Educationa Testing Service, Princeton, New Jersey, Besco, Robert 0. The measurement and prediction of success in graduate schoo. Ph.D. dissertation, Purdue University, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Borg, Water R. GRE aptitude scores as predictors of GPA for graduate students in education. Educationa and Psychoogica Measurement, 1963, 23 (2): Capps, Marian P., and Decosta, Frank A. Contributions of the Graduate Record Examinations and the Nationa Teacher Examinations to the prediction of graduate schoo success. Journa of Educationa Research, 1957, 50 (5): Cark, Henry. Graduate Record Examination correations with grade-point averages in the Department of Education at Northern Iinois University, Unpubished Master s thesis, Northern Iinois University, June

34 Conway, Sister Madona Therese. The reationship of the Graduate Record Examination resuts to achievement in the Graduate Schoo at the University of Detroit. Unpubished Master s thesis, University of Detroit, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Creager, John A. A study of graduate feowship appicants in terms of Ph.D. attainment. Technica Report No. 18, Office of Scientific Personne, Nationa Academy of Sciences- Nationa Research Counci, Washington, D.C., Creager, John A. Predicting doctorate attainment with GRE and other variabes. Technica Report No. 25, Office of Scientific Personne, Nationa Academy of Sciences -Nationa Research Counci, Washington, D.C., Dawes, Robyn M. A case study of graduate admissions: Appication of three principes of human decision making. American Psychoogist, 1971, 26 (2): Duff, Frankin L., and Aukes, Lewis E. The reationship of the Graduate Record Examination to success in the Graduate Coege (a suppementary comparative anaysis of eight previousy reported studies). Bureau of Institutiona Research and Office of Instructiona Research, University of Iinois, October Eckhoff, Constance M. Predicting graduate success at Winona State Coege. Educationa and Psychoogica Measurement, 1966, 26 (2) : Ewen, Robert B. The GRE psychoogy test as an unobtrusive measure of motivation. Journa of Appied Psychoogy, 1969, 53 (5): Forida State University, Office of Academic Research and Panning. The prediction of grade-point average in graduate schoo at the Forida State University, Parts I & II. Forida State University, December Forida State University, Office of Institutiona Research and Service. Reationship between Graduate Record Examinations Aptitude Test scores and academic achievement in the Graduate Schoo at Forida State University. Forida State University, Hackman, J. Richard, Wiggins, Nancy, and Bass, Aan R. Prediction of ong-term success in doctora work in psychoogy. Educationa and Psychoogica Measurement, 1970, 30: Hansen, W. Lee. Prediction of graduate performance in economics. Department of Economics, University of Wisconsin, Apri 1970 (Mimeographed). 28

35 Harvey, Phiip R. Predicting graduate schoo performance in education. Unpubished ETS report, Reported by G.V. LannhominGRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, King, Donad C., and Besco, Robert 0. The Graduate Record Examination as a seection device for graduate research feows. Educationa and Psychoogica Measurement, 1960, 20 (4): Lannhom, Gerad V., Marco, Gary L., and Schrader, Wiiam B. Cooperative studies of predicting graduate schoo success. GRE Specia Report Educationa Testing Service, Princeton, New Jersey, August Law, Aexander. The prediction of ratings of students in a doctora training program. Educationa and Psychoogica Measurement, 1960, 20 (4): Lorge, Irving. Reationship between Graduate Record Examinations and Teachers Coege, Coumbia University, Doctora Verba Examinations. Letter to G.V. Lannhom, dated September 21, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Madaus, George F., and Wash, John J. Departmenta differentias in the predictive vaidity of the Graduate Record Examinations Aptitude Tests. Educationa and Psychoogica Measurement, 1965, 25 (4): Mehrabian, Abert. Undergraduate abiity factors in reationship to graduate performance. Educationa and Psychoogica Measurement, 1969, 29 (2): Michae, Wiiam B., Jones, Robert A., A-Amir, Hudhai, Puias, Cavin M., Jackson, Miche, and Goo, Vaerie. Correates of a pass-fai decision for admission to candidacy in a doctora program. Educationa and PsychoogicaMeasurement, 1971, 31 (4): Michae, Wiiam B., Jones, Robert A., and Gibbons, Biie D. The prediction of success in graduate work in chemistry from scores on the Graduate Record Examinations. Educationa and Psychoogica Measurement, 1960, 20 (4): Newman, Richard J. GRE scores as predictors of GPA for psychoogy graduate students. Educationa and Psychoogica Measurement, 1968, 28 (2):

36 Office of Educationa Research. Study of GRE scores of geoogy students matricuating in the years RP-Abstract, Yae University, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Osen, Marjorie. The predictive effectiveness of the Aptitude Test and the Advanced Bioogy Test of the GRE in the Yae Schoo of Forestry. Statistica Report 55-6, Educationa Testing Service, Princeton, New Jersey, out of print. Roberts, Pamea T. An anaysis of the reationship between Graduate Record Examination scores and success in the Graduate Schoo of Wake Forest University. Master s thesis, Wake Forest University, August Robertson, Macom, and Niesen, Winnifred. The Graduate Record Examination and seection of graduate students. American Psychoogist, 1961, 16 (10): Robinson, Donad W. A comparison of two batteries of tests as predictors of first year achievement in the graduate schoo of Bradey University. Ph.D. dissertation, Bradey University, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Rock, Donad A. The prediction of doctorate attainment in psychoogy, mathematics, and chemistry (GRE Board Preiminary Report). Princeton, N. J.: Educationa Testing Service, August 1972 (ERIC Document Reproduction Service No. ED ). Later pubished as GRE Board Research Report 69-6aR, June Roscoe, John T., and Houston, Samue R. The predictive vaidity of GRE scores for a doctora program in education. Educationa and Psychoogica Measurement, 1969, 29 (2): Sacramento State Coege, Test Office. An anaysis of traditiona predictor variabes and various criteria of success in the Master s degree program at Sacramento State Coege for an experimenta group who received Master s degrees in the spring 1968, and a comparabe contro group who withdrew from their programs. Test Office Report 69-3, Sacramento State Coege, October Shaffer, Juie, and Rosenfed, Howard. MAT-GREprediction study-initia resuts. Intradepartmenta memorandum, Department of Psychoogy, University of Kansas, March

37 Sistrunk, Francis. The GREs as predictors of graduate schoo success in psychoogy. Letter to G.V. Lannhom, dated October 3,196. Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Seeper, Midred L. Reationshipof scores on the Graduate Record Examination to grade point averages of graduate students in occupationa therapy. Educationa and Psychoogica Measurement, 1961, 21 (4): Tuy, G. Emerson. Screening appicants for graduate study with the Aptitude Test of the Graduate Record Examinations. Coege and University, 1962, 38: University of Virginia, Office of Institutiona Anaysis. Correations between admissions criteria and University of Virginia grade-point averages, Graduate Schoo of Arts and Sciences, Fa University of Virginia, circa (Mimeographed) Waace, Anita D. The predictive vaue of the Graduate Record Examinations at Howard University. Unpubished Master s thesis, Howard University, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, White, Eizabeth L. The reationship of the Graduate Record Examinations resuts to achievement in the Graduate Schoo at the University of Detroit. Unpubished Master s thesis, University of Detroit, Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, White, Gordon Wiiam. A predictive vaidity study of the Graduate Record Examinations Aptitude Test at the University of Iowa. Unpubished Master s thesis, University of Iowa, June Reported by G.V. Lannhom in GRE Specia Report 68-1, Educationa Testing Service, Princeton, New Jersey, Wiiams, John D., Harow, Steven D., and Grab, De. A ongitudina study examining prediction of doctora success: Grade-point average as criterion, or graduation vs. non-graduation as criterion. Journa of Educationa Research, December 1970, 64 (4):