2015 Association of American Medical Colleges. May not be reproduced or distributed without prior written permission.

Transcription

1

2 2015 Association of American Medical Colleges. May not be reproduced or distributed without prior written permission. This is a publication of the Association of American Medical Colleges. The AAMC serves and leads the academic medicine community to improve the health of all. June 2015 (ver. 2)

3 Dear Admissions Officers: The new Medical College Admission Test (MCAT ) was introduced in April The new MCAT exam is designed for today s medical students and tomorrow s doctors. It tests the knowledge and skills that students need when they enter medical school. The transition to the new MCAT exam coincides with your admissions cycle for the 2016 entering class. Transitioning to the new MCAT exam will present new opportunities and challenges to your admissions committee. When the 2016 application cycle opens on July 1, you will receive scores from applicants who took the new exam, as well as scores from applicants who took the old exam. We expect about half of your applicants will report scores from the new MCAT exam and half will report scores from the old MCAT exam. As a result, this guide provides the information and data you need to work with both new and old MCAT scores. The first part of this guide focuses on the interpretation and use of scores from the new MCAT exam. The second part of this guide focuses on the interpretation and use of scores from the old MCAT exam. The AAMC will be available every step of the way to help you make the transition from using scores from the old MCAT exam to using scores from the new MCAT exam. Please don t hesitate to contact us if you have questions. You can reach us by at mcat2015@aamc.org. Sincerely, Cynthia A. Searcy Senior Director, MCAT Research and Development i

4 Using MCAT Data in in Medical Student Selection Table of Contents Part 1: Using Scores from the New MCAT Exam in Student Selection... 1 The New Medical College Admission Test (MCAT)... 2 Benefits of the New MCAT Exam... 5 Content of the New MCAT Exam... 7 Biological and Biochemical Foundations of Living Systems...9 Chemical and Physical Foundations of Biological Systems...10 Psychological, Social, and Biological Foundations of Behavior...11 Critical Analysis and Reasoning Skills...13 Scores from the New MCAT Exam Using Percentile Ranks in 2016 Admissions...16 Using Confidence Bands in 2016 Admissions...19 Using Score Profiles in 2016 Admissions...20 Making Decisions about Applicants in the 2016 Admissions Cycle...21 Designing Thresholds for Screening and Interviewing...21 Making Decisions about Individual Applicants...22 Data about the New MCAT Exam Timeline for Reporting Data about the New Exam...23 MCAT Validity Research Agenda...23 The Psychological, Social, and Biological Foundations of Behavior Validity Study...24 The Longitudinal MCAT Validity Study...25 National MCAT Validity Data...25 References Appendix A: The MCAT Psychological, Social, and Biological Foundations of Behavior Validity Study Results Appendix B: Sample Questions for the New MCAT Exam Part 2: Using Scores from the Old MCAT Exam in Student Selection Content of the Old MCAT Exam Scores from the Old MCAT Exam Validity Data on the Old MCAT Exam References ii

5 Part 1: Using Scores from the New MCAT Exam in Student Selection 1

6 The New Medical College Admission Test (MCAT) The new MCAT exam is designed for today s medical students... and tomorrow s doctors. There is no question that the nation s health care system is changing and future physicians will need more and better tools to manage the delivery system and meet the needs of the nation s changing, growing, aging population. Medical schools are changing their curricula to meet these challenges by targeting the competencies that tomorrow s physicians will need. The new MCAT exam, introduced in April 2015, was designed to match the adoption of competency-based curricula in medical schools by testing foundational knowledge and skills that students need when they enter medical school. The transition to the new MCAT exam coincides with the 2016 application cycle. Transitioning to a new MCAT exam will present challenges to you and your admissions committee, but it also will present opportunities to consider a wider range of applicants with a broader range of academic competencies. This guide is designed to help you use MCAT scores during this transition. It places the new exam in the broader context of your individualized, holistic review of applicants experiences, attributes, and academic preparation to admit students who best fit your school s mission and goals. It describes the foundational concepts and skills in the natural, behavioral, and social sciences that are tested by the new exam, making connections between pre-medical preparation and the medical school curriculum. It describes ways to attach meaning to scores from the new exam, highlighting important changes in the score scales and score reports intended to help you make better decisions about the applicants who will best fit your medical school curriculum. It acknowledges the complexities of considering applicants in this admissions cycle during which half of your applicants will report scores from the old exam and half will report scores from the new exam providing guidance on ways to think about MCAT scores in this context. It concludes with a timeline and sources of data to help you learn more about the ways the new exam will support your admissions decision making. Holistic review practices can help you transition from the old MCAT exam to the new exam. Holistic review practices provide a firm foundation for selecting applicants with the academic and personal competencies that future physicians need. Your institutional mission, goals, and priorities provide a framework for evaluating applicants. They lay the foundation for admitting a class of capable, caring students who bring diverse interests, talents, and experiences to your institution. Applicants provide admissions committees with rich information about their experiences, attributes, and academic backgrounds through their applications, personal statements, and interviews. Letter writers also provide information about applicants academic and personal competencies. Holistic review practices provide an important foundation for transitioning to the new exam. With the introduction of the new MCAT exam, it is important to remember the following best practices for considering data about academic preparation in the context of the varied information collected during the admissions process. As you did for the old MCAT exam, you should triangulate score information from the new MCAT exam with information about applicants course completion, grades, grade trends, institutional selectivity, research experience, and other academic indicators. You should look for consistencies and inconsistencies in the stories these data tell. 2

7 Your admissions committee should carefully consider the rich and varied data that applicants provide. Table 1 describes the results of a 2013 survey about the importance of academic metrics, experiences, demographics, and other sources of applicant information for deciding which applicants to interview and which to admit. The table highlights the importance of different types of data in admissions decision making. Consideration of these data will help your admissions committee construct a class that meets the academic, clinical, service, and research missions of your medical school. Table 1. Mean Importance Ratings of Academic, Experiential, and Demographic Application Data Used by Admissions Committees for Making Decisions about Which Applicants to Receive an Interview Invitation and Offer Acceptance (N=127) 1 Mean Importance Ratings Highest Importance Ratings (> 3.0) Academic Metrics Experiences Demographics 2 Sources of Applicant Information UGPA: Cumulative science/math MCAT total score Upward or downward grade trend UGPA: Cumulative total Performance in a postbaccalaureate program Selectivity of undergraduate institution (Private) 5 Healthcare experience Community service/ volunteer experience Experience with underserved populations Navigated through cultural barriers or challenges Leadership experience U.S. citizenship/ permanent residency (Public) 5 State residency (Public) 5 Interview assessments 3 Letters of evaluation or recommendation Personal statements Medium Importance Ratings (>2.5 and <3.0) On schedule to meet pre-medical coursework UGPA: Cumulative nonscience/math Research experience (Private) 5 Experience with populations unlike the applicant Lack of access to optimal educational resources Special family obligations or other circumstances Work or athletic scholarship obligations while in school Rural or urban background (Public) 5 First-generation college student U.S. citizenship/ permanent residency (Private) 5 Race/ethnicity Socioeconomic status (SES) Secondary application responses 4 Lowest Importance Ratings (<2.5) Completion of challenging non-science courses Selectivity of undergraduate institution (Public) 5 Experience with prejudice Research experience (Public) 5 Rural or urban background (Private) 5 Multilingual Legacy Gender State residency (Private) 5 1 Importance was rated on a four-point scale ranging from 1 to 4 ("Not Important," "Somewhat Important," "Important," and "Very Important," respectively). For each variable, we computed an overall mean importance rating based on admissions officers ratings of importance for making decisions about whom to interview and whom to accept (the mean importance rating for the interview variable is the exception to this rule because interview data were not available until applicants were invited to interview). We chose to classify variables using overall mean importance ratings because their mean importance ratings were similar for the interview and the acceptance phases. Variables are ordered by overall mean importance rating. 2 The demographic variables listed here were culled from a larger list of variables that included personal attributes such as "Service Orientation," "Integrity," "Communication Skills," and "Cultural Competence," for example. 3 Only available at the admissions stage where admissions committees make a decision to offer an acceptance. 4 Only available for institutions that use application data to select the applicants who will receive secondary applications. 5 Overall mean importance ratings for public and private institutions were significantly different from one another. 3

8 As shown in Table 2, national-level data on acceptance rates for applicants who took the old MCAT exam between 2012 and 2014 reinforce the messages these 2013 survey data provide. Each year, some applicants with high MCAT scores and undergraduate GPAs are rejected by all the medical schools to which they applied. In contrast, other applicants with more modest MCAT scores and undergraduate GPAs are accepted by at least one medical school. While undergraduate GPAs and MCAT total scores are important factors in admissions, they are not the sole determinants of admissions decisions. In next year s version of this guide, you are likely to see a similar pattern of results for applicants who took the new MCAT exam. Scores from the new exam should not outweigh other application data in deciding which applicants will get secondary invitations, interview invitations, or acceptances. Table 2. Percentage of Applicants Accepted into at Least One Medical School, by Old MCAT Total Score and Undergraduate GPA Range Old MCAT Total GPA Total All % 2% 11% 21% 38% 61% 76% 84% 88% 91% 69% 3/80 4/162 57/ /1,526 1,363/3,554 4,232/6,978 7,110/9,361 6,298/7,504 3,688/4,176 1,385/1,519 24,457/35, % 2% 8% 16% 27% 45% 64% 76% 80% 85% 52% 0/177 8/367 83/1, /2,332 1,332/4,866 3,725/8,284 5,997/9,359 4,513/5,973 1,782/2, /514 18,246/35, % 2% 5% 12% 22% 32% 48% 62% 70% 75% 36% 1/336 13/553 67/1, /2,607 1,010/4,691 2,307/7,151 3,600/7,455 2,382/3, /1, /234 10,689/29, % 1% 4% 11% 18% 23% 35% 47% 58% 65% 25% 0/370 5/561 41/1, /2, /3,344 1,012/4,369 1,453/4, /1, /547 74/113 4,643/18, % 0% 3% 8% 17% 19% 29% 39% 48% 53% 18% 0/388 2/553 25/ /1, /2, /2, /1, / /233 21/40 1,954/10, % 1% 3% 6% 12% 16% 24% 27% 26% 29% 12% 0/368 4/386 19/626 54/ /1, / /746 85/310 22/86 7/24 660/5, % 0% 3% 5% 9% 15% 21% 28% 32% 30% 9% 0/274 1/284 12/355 24/486 47/512 57/388 59/276 33/117 15/47 3/10 251/2, % 0% 1% 3% 9% 14% 17% 16% 18% -- 6% 0/196 0/151 2/179 8/240 19/221 22/152 18/109 6/37 3/17 79/1, % 0% 0% 0% 8% 12% 15% 7% % 0/132 0/77 0/94 0/91 7/88 8/68 6/39 1/14 22/ % 0% 0% 0% 0% 14% 18% -- 2% 0/53 0/40 0/42 0/28 0/30 2/14 2/11 4/ % -- 0% 0% % 0/42 0/10 0/12 0/90 All 0% 1% 5% 12% 24% 39% 57% 71% 79% 85% 44% 4/2,419 37/3, /6,229 1,460/12,070 4,887/20,601 11,978/30,769 18,954/33,316 14,520/20,521 6,757/8,522 2,102/2,459 61,005/140,049 Notes: 1. Dark green shading = acceptance rates 75%; Light green shading = acceptance rates of 50 74%; grey shading = acceptance rates of 25 49%. 2. Dashes = cells with fewer than 10 observations; blank cells = cells with 0 observations. 3. For students who took the old MCAT exam multiple times, the most recent old MCAT total score in each application year was used in this analysis. 4

9 Benefits of the New MCAT Exam The new exam tests the academic preparation students need for today s medical school. The new MCAT exam is designed to help medical school admissions committees select students who are academically prepared for the curriculum they will begin in The blueprints for the new exam are evidence-based and target concepts that medical school faculty, residents, and medical students recently rated as important to entering students success. They are organized around the academic competencies described by seminal reports like the Scientific Foundations for Future Physicians ( and the Behavioral and Social Science Foundations for Future Physicians ( socialsciencefoundations). The quantitative and qualitative research that supports the definition and development of the new MCAT exam is strong (Schwartzstein et al. 2013). The new exam shifts the focus from testing what applicants know to testing how well they use what they know. It focuses on the outcomes of learning by asking applicants to apply what they have learned. The new exam asks applicants to use their knowledge of the natural, behavioral, and social sciences to solve problems that call for scientific reasoning. The new exam asks applicants to show that they can think and learn like physicians. In today s information age, it is no longer possible for physicians to memorize all the information they need to make informed clinical decisions. Future physicians will need to know where to find credible information, how to evaluate it, and how to translate it into high-quality care. The new MCAT blueprints recognize this by emphasizing scientific reasoning and analysis skills, asking applicants to reason about research designs and results, and asking applicants to interpret data and draw conclusions from them. The new exam underscores the important roles of behavioral and sociocultural factors in health and illness by including a section that tests concepts from the behavioral and social sciences. This foundational knowledge will prepare medical students to learn how behavior interacts with biological factors to influence health outcomes and how social inequities affect patients health. The new exam communicates the value that medical school admissions committees place on broad preparation for medical school, and it reinforces the diversity of interests and preparation that committee members look for in their applicants. Finally, the new exam recognizes the heavy information-processing requirements of medical school curricula and asks students to demonstrate that they understand and can critically analyze text. The new exam provides more reliable scores, supporting better decisions. The architects of the new MCAT exam understood that medical schools need reliable information about applicants academic preparation. In response, they designed each section of the new exam with more test questions than were included on the old exam. More questions per section provide more information about applicants. The new scores are estimated with greater precision. Improved precision means improved decision making, allowing you to consider carefully applicants strengths and weaknesses across the four sections of the new exam as you judge your applicants fit with your curriculum. The new exam was designed with fairness in mind. Concerns about fairness played a critical role in developing the new exam blueprints, and particularly in ensuring that the concepts and skills tested on the new exam are taught widely. The architects of the new exam checked course offerings at minority-serving and under-resourced colleges and universities to ensure applicants have access to the needed courses. The designers also surveyed faculty at these institutions to learn about the content of their courses, and they examined data on applicants course-taking behaviors to learn whether applicants at these institutions complete the needed classes. These 5

10 analyses of course offerings, course content, and course taking at minority-serving and under-resourced institutions guided the blueprints for the new exam. Another prominent fairness issue in designing the new exam was testing time. Although each section of the new exam includes more questions than sections of the old exam, the design of the new test gives examinees more working time per question than the old exam. Applicants have more time to review passages, read questions, and decide on their answers. Finally, as previously mentioned, the new MCAT exam balances testing in the natural sciences with testing in the behavioral and social sciences and critical analysis and reasoning. The hope is that this balance will help diversify the physician workforce by making the exam and medical school application more attractive to individuals from more varied academic and demographic backgrounds. 6

11 Content of the New MCAT Exam This section of the guide provides information about the design, interpretation, and use of the new MCAT exam. For ease of reading, this guide describes MCAT sections that have similar content and format together. The new MCAT exam has four sections: 1. Biological and Biochemical Foundations of Living Systems 2. Chemical and Physical Foundations of Biological Systems 3. Psychological, Social, and Biological Foundations of Behavior 4. Critical Analysis and Reasoning Skills The new and old MCAT exams are different. To be ready for medical school in 2016, applicants need preparation in biology, chemistry, physics, and verbal reasoning, just as they did in the past. The new MCAT exam tests these concepts and also tests preparation in first-semester biochemistry, psychology, and sociology, with a greater emphasis on demonstrating scientific reasoning skills and by capitalizing on recent research about information-processing skills. Content New MCAT Exam Old MCAT Exam Biology Chemistry Physics Verbal Reasoning Biochemistry Psychology Sociology The new exam: Requires broader preparation Asks students to apply what they ve learned Focuses on behavioral and sociocultural aspects of health Incorporates the latest science on information processing The two natural sciences and the behavioral and social sciences sections of the new MCAT exam test 10 foundational competencies and four scientific inquiry and reasoning skills. These sections ask students to combine their knowledge of concepts that are foundational for learning in medical school with their inquiry and reasoning skills. As shown in the figure on the following page, MCAT questions test foundational concepts and scientific reasoning skills together. The resulting scores provide information about applicants readiness to learn in medical school. 7

12 Foundational Concepts Tested on the New MCAT Exam Biological and Biochemical Foundations of Living Systems Chemical and Physical Foundations of Biological Systems Psychological, Social, and Biological Foundations of Behavior Foundational Concept 1 Foundational Concept 2 Foundational Concept 3 Foundational Concept 4 Foundational Concept 5 Foundational Concept 6 Foundational Concept 7 Foundational Concept 8 Foundational Concept 9 Biomolecules have unique properties that determine how they contribute to the structure and function of cells, and how they participate in the processes necessary to sustain life. Highly organized assemblies of molecules, cells, and organs interact to carry out the function of living organisms. Complex systems of tissues and organs sense the internal and external environments of multicellular organisms, and through integrated functioning, maintain a stable internal environment within an ever-changing external environment. Complex living organisms transport materials, sense their environment, process signals, and respond to changes using processes that can be understood in terms of physical principles. The principles that govern chemical interactions and reactions form the basis for a broader understanding of the molecular dynamics of living systems. Psychological, sociocultural, and biological factors influence the ways that individuals perceive, think about, and react to the world. Psychological, sociocultural, and biological factors influence behavior and behavior change. Psychological, sociocultural, and biological factors influence the way we think about ourselves and others. Cultural and social differences influence well-being. Ways that students use their scientific inquiry and reasoning skills to solve problems on the exam: Demonstrating understanding of scientific concepts and principles Identifying the relationships between closely related concepts Reasoning about scientific principles, theories, and models Analyzing and evaluating scientific explanations and predictions Demonstrating understanding of important components of scientific research Reasoning about ethical issues in research Interpreting patterns in data presented in tables, figures, and graphs Reasoning about data and drawing conclusions from them Foundational Concept 10 Social stratification and access to resources influence well-being. 8

13 Biological and Biochemical Foundations of Living Systems Medical school applicants must be prepared to learn about the biological and biochemical concepts that contribute to health and disease. This section tests applicants academic preparation in three foundational concepts that are building blocks for learning in medical school about how: The major biochemical, genetic, and molecular functions of the cell support health and lead to disease Cells grow and integrate to form tissues and organs that carry out essential biochemical and physiological functions The body responds to internal and external stimuli to support homeostasis and the ability to reproduce What This Section Tests The Biological and Biochemical Foundations of Living Systems section asks examinees to solve problems by combining their knowledge of foundational concepts from biology, biochemistry, general chemistry, and organic chemistry with their scientific inquiry and reasoning skills. This new section aligns most closely with the Biological Sciences section of the old MCAT exam. The table below lists the foundational concepts and content categories tested in this section. It also provides examples of the ways examinees are asked to combine their knowledge of foundational concepts with their scientific reasoning skills to answer test questions on the Biological and Biochemical Foundations of Living Systems section. Biological and Biochemical Foundations of Living Systems Foundational Concept 1 Biomolecules have unique properties that determine how they contribute to the structure and function of cells, and how they participate in the processes necessary to sustain life. Content Categories Structure and functions of protein and their constituent amino acids Transmission of genetic information from the gene to the protein Transmission of heritable information from generation to generation and the processes that increase genetic diversity Principles of bioenergetics and fuel molecule metabolism Foundational Concept 2 Highly organized assemblies of molecules, cells, and organs interact to carry out the function of living organisms. Content Categories Assemblies of molecules, cells, and groups of cells within singular cellular and multicellular organisms The structure, growth, physiology, and genetics of prokaryotes and viruses Processes of cell division, differentiation, and specialization Foundational Concept 3 Complex systems of tissues and organs sense the internal and external environments of multicellular organisms, and through integrated functioning, maintain a stable internal environment within an ever-changing external environment. Content Categories Structure and functions of the nervous and endocrine systems and ways in which the systems coordinate the organ systems Structure and integrative functions of the main organ systems Questions in this section of the test ask examinees to combine their knowledge of the foundational concepts listed above with their scientific inquiry and reasoning skills. Below are examples: Recalling the structural characteristics of two tissues and relating them to one another Applying their understanding of Le Châtelier s Principle to explain differences in deprotonation of organic acids when added to blood versus pure water Using knowledge of adaptive immune response to evaluate the acceptability of a treatment for use in a clinical context Forming a hypothesis about the effect of the pineal gland on thermogenesis based on the data from an experiment investigating the interaction of temperature and pineal gland activity on body and organ weights for hamsters under different experimental conditions Using data about wavelength and light absorption to determine the color perception of an individual with a given phenotype 9

14 Chemical and Physical Foundations of Biological Systems Medical school applicants must be prepared to learn about the mechanical, physical, and biochemical functions of human tissues, organs, and organ systems and how these contribute to health and disease. This section tests applicants preparation in two foundational concepts that are building blocks for learning in medical school about: The physiological functions of the respiratory, cardiovascular, and neurological systems in health and disease Molecular and cellular functions in health and disease What This Section Tests The Chemical and Physical Foundations of Biological Systems section asks test takers to solve problems by combining their knowledge of foundational concepts from biology, biochemistry, physics, and general and organic chemistry with their scientific inquiry and reasoning skills. This section aligns most closely with the Physical Sciences section of the old MCAT exam. The table below lists the foundational concepts and content categories tested in this section. It also provides examples of the ways examinees are asked to combine their knowledge of foundational concepts with their scientific inquiry and reasoning skills to answer test questions on the Chemical and Physical Foundations of Biological Systems section. Foundational Concept 4 Complex living organisms transport materials, sense their environment, process signals, and respond to changes using processes that can be understood in terms of physical principles. Content Categories Translational motion, forces, work, energy, and equilibrium in living systems Importance of fluids for the circulation of blood, gas movement, and gas exchange Electrochemistry and electrical circuits and their elements How light and sound interact with matter Atoms, nuclear decay, electronic structure, and atomic chemical behavior Chemical and Physical Foundations of Biological Systems Foundational Concept 5 The principles that govern chemical interactions and reactions form the basis for a broader understanding of the molecular dynamics of living systems. Content Categories Unique nature of water and its solutions Nature of molecules and intermolecular interactions Separation and purification methods Structure, function, and reactivity of biologically relevant molecules Atoms, nuclear decay, electronic structure, and atomic chemical behavior Questions in this section of the test ask examinees to combine their knowledge of the foundational concepts listed above with their scientific inquiry and reasoning skills. Below are examples: Identifying the relationship between the distribution of electric charges in the axon and the electric field lines they produce Recognizing the principles of flow characteristics of blood in the human body and applying the appropriate mathematical model to an unfamiliar scenario Changing the experimental conditions of a test for proteins in a solution to prevent the formation of precipitates Selecting between the standard and Doppler ultrasound techniques for a given context, considering the appropriateness, precision, and accuracy of each technique Using, analyzing, and interpreting data in a graph to determine the half-life of a radioactive substance used to measure cardiac function 10

15 Psychological, Social, and Biological Foundations of Behavior Medical school applicants must be prepared to learn about the impact of behavioral and sociocultural factors on illness and health outcomes. This section tests examinees academic preparation in five foundational concepts in psychology, sociology, and biology that provide the building blocks for learning in medical school about the ways: Cognitive and perceptual processes influence the understanding of health and illness Behavior can either support health or increase risk for disease Perception, attitudes, and beliefs influence interactions with patients and other members of the health care team Patients social and demographic backgrounds influence their perceptions of health and disease, the health care team, and therapeutic interventions Social and economic factors can affect access to care and the probability of maintaining health and recovering from disease What This Section Tests The Psychological, Social, and Biological Foundations of Behavior section tests academic preparation in the behavioral and social sciences. Prior versions of the MCAT exam have not tested this content. This section tests the foundational concepts in psychology, sociology, and biology that tomorrow s doctors need to serve an increasingly diverse population and have a clear understanding of the impact of behavior and sociocultural differences on health. Like the natural sciences sections, this section asks test takers to solve problems by combining their knowledge of foundational concepts with their scientific inquiry and reasoning skills. It does not measure applicants interpersonal skills, the way they will behave, or their attitudes and beliefs about social issues. The table below lists the foundational concepts tested on this section. It also provides examples of the ways examinees are asked to combine their knowledge of foundational concepts with their scientific inquiry and reasoning skills to answer test questions on the Psychological, Social, and Biological Foundations of Behavior section. Foundational Concept 6 Psychological, sociocultural, and biological factors influence the ways that individuals perceive, think about, and react to the world. Content Categories Sensing the environment Making sense of the environment Responding to the world Psychological, Social, and Biological Foundations of Behavior Foundational Concept 7 Psychological, sociocultural, and biological factors influence behavior and behavior change. Content Categories Individual influences on behavior Social processes that influence human behavior Attitude and behavior change Foundational Concept 8 Psychological, sociocultural, and biological factors influence the way we think about ourselves and others. Content Categories Self-identity Social thinking Social interactions Foundational Concept 9 Cultural and social differences influence well-being. Content Categories Understanding social structure Demographic characteristics and processes Foundational Concept 10 Social stratification and access to resources influence well-being. Content Categories Social inequity Questions in this section of the test ask examinees to combine their knowledge of foundational concepts listed above with their scientific inquiry and reasoning skills. Below are examples: Drawing conclusions about the type of memory affected by an experimental manipulation when shown a graph of findings from a memory experiment Reasoning about whether a causal explanation is possible when given an example of how personality predicts individual behavior Distinguishing the kinds of claims that can be made when using longitudinal data, cross-sectional data, or experimental data in studies of social interaction Identifying the relationship between demographic variables and health variables reported in a table or figure Identifying the relationship between social institutions that is suggested by an illustration used in a public health campaign 11

16 Do scores from the Psychological, Social, and Biological Foundations of Behavior section add value to admissions decisions? Eleven medical schools are partnering to study the Psychological, Social, and Biological Foundations of Behavior section. More than 2,000 medical students from these schools took a prototype version of this section before the start of the fall 2013 semester. Schools then collected participants grades in medical school courses related to the behavioral and social sciences. Scores were correlated with participants grades in these courses. These early results show that scores from the Psychological, Social, and Biological Foundations of Behavior section predict medical students grades in courses such as Foundations of Psychiatric Medicine, Neuroscience, and Epidemiology/Public Health and predict grades in these courses better than scores from the old MCAT exam. These results suggest that these scores add value to student selection decisions. Refer to Appendix A for more information about the study and results. 12

17 Critical Analysis and Reasoning Skills The Critical Analysis and Reasoning Skills section has a different organizing structure than the other sections of the exam. It tests how well test takers comprehend, analyze, and evaluate what they read, draw inferences from text, and apply arguments to new ideas and situations. It asks applicants to process information, draw conclusions, and solve problems from information that is presented in passages. This section aligns most closely with the Verbal Reasoning section of the old MCAT exam. This section tests examinees ability to process information by reading passages from a diverse set of disciplines in the humanities and social sciences. However, the questions on the Critical Analysis and Reasoning Skills test do not rely on specific background knowledge in the humanities and social sciences. Applicants get all the information they need to answer the questions in the accompanying passages or in the questions themselves. The new exam gives increased attention to passages about population health, studies of diverse cultures, ethics, and philosophy to encourage applicants to read broadly and familiarize themselves with the issues and arguments these disciplines raise. Questions in this section of the test ask examinees to demonstrate their information-processing skills by: Understanding the basic components of the text Inferring meaning from rhetorical devices, word choice, and text structure Integrating different components of the text to increase comprehension Applying or extrapolating ideas from the passage to new contexts Assessing the impact of introducing new factors, information, or conditions to ideas from the passage 13

18 Scores from the New MCAT Exam This section of the guide provides information about interpreting scores from the new exam. It contains information about the new score scale and use of percentile ranks, confidence bands, and score profiles to make decisions about applicants in the 2016 application cycle. Scores are reported on different scales because the new and old exams test different things. The new score scales were developed to have nice round, memorable numbers in the center and scales that can t be confused with the scales from the old exam, percent-correct scales, or other commonly used score scales. The four section scores are centered at 125 and range from 118 to 132. Scores from the four sections are summed to produce a total score that is centered at 500 and ranges from 472 to 528. The new scores draw attention to the center of the scales and the top half of the distribution to encourage admissions committees to consider applicants with a wider range of scores than they have in the past. On the old exam, the average total score for accepted applicants was 31 and ranged from 3 to 45. Many selection decisions focused on the top third of the old score scale, but research on the old exam suggests that the students who enter medical school with scores at the center of the scale succeed; they graduate in four or five years and pass their licensing exams on the first try (Dunleavy et al. 2013). Section Scores Sum to the Total Score = Chemical and Physical Foundations of Biological Systems Critical Analysis and Reasoning Skills Biological and Biochemical Foundations of Living Systems Psychological, Social, and Biological Foundations of Behavior Total Score

19 The new scores are more reliable. As we have already mentioned, the new section scores are more reliable than the section scores from the old exam because the new exam has more questions per section. They provide more precise information about test takers academic preparation and provide better pictures of examinees strengths and weaknesses across the four test sections. The new score report provides more information. You will receive a lot of information about applicants preparation from the new score reports, which describe applicants scores using percentile ranks, confidence bands, and score profiles. This is what the new score report looks like: 15

20 Using Percentile Ranks in 2016 Admissions Because the new exam uses new score scales, admissions committees will essentially have to start over, attaching meaning to the scores and developing new conventions for selecting students. To help with this process, the new score reports will provide percentile ranks one for each section score and one for the total score. In the beginning, the percentile ranks will provide the clearest information about applicants work on the new exam. The percentiles show how individual test takers compare with the population of test takers who prepared for and took the new exam. They represent the percentage of examinees at or below a given score on the exam. Scores with higher percentile ranks represent higher levels of performance on the exam. As you gain experience working with new MCAT scores, you will begin to attach meaning to new scores (just as scores from the old test have meaning for you) by observing the academic performance of students admitted with different MCAT scores. You will begin to see which scores spell success in your program. As you gain this experience, you are likely to start using the scores more than the percentile ranks. Eventually, you will find that confidence bands and score profiles provide helpful information about the scores themselves. We will talk about that next. Developing Percentile Rank Tables for the 2015 Testing Year The percentile ranks reported for the 2015 testing year are based on the scores of examinees who took the exam on the two April and May test dates (Figure 1). After careful examination of the April and May 2015 scores and based on historic MCAT data, the AAMC made adjustments so that the distribution of scores reported in these percentile rank tables reflects the best estimate of the score distribution that would result in a typical testing year. The AAMC carefully developed these adjustments using a wide range of data about past examinees. These percentile rank tables will allow you to compare the performance of April and May examinees with the performance of other 2015 examinees. The AAMC will use these same percentile ranks for all the scores reported in the 2015 testing year that is, all scores from the 2015 testing year will be attached to the same set of percentile ranks. 16

21 Figure 1. Summary of total and section scores from the MCAT exam based on results for tests administered in April and May MCAT Total Score 5 4 Percent MCAT Total Score MCAT Total Percentile MCAT Total Score Score Rank Mean = Std. Deviation = AAMC would result in a typical testing year For example, 77 percent of MCAT Notes The percentile ranks in these tables are based on MCAT results from examinees who tested in April and May After careful analysis of the April and May 2015 scores and based on historic MCAT data, the made adjustments so that the distribution of scores reported in these tables reflects the best estimate of the score distribution that The AAMC carefully developed these adjustments using a wide range of data about past examinees. The column labeled Percentile Rank provides the percentage of results in a typical testing year equal to or less than each score point. 1 total scores are likely to be equal to or less than 508 across all exams administered in a typical year. 1 Prior versions of these tables (pre 2012), as well as current score reporting systems maintained by AMCAS, report the "Percentile Rank Range" in these columns rather than a single percentile rank. The percentile rank columns in the current tables correspond to the upper bound of the percentile rank ranges provided in these other versions. The lower bounds in these other versions essentially serve to provide continuity across score points. (continued on next page) 17

22 Figure 1. Summary of total and section scores from the MCAT exam based on results for tests administered in April and May (Continued) Chemical and Physical Foundations of Biological Systems Percent Chemical and Physical Foundations of Biological Systems Section Score Critical Analysis and Reasoning Skills Percent Critical Analysis and Reasoning Skills Section Score Section Percentile Chemical and Physical Score Rank Foundations of Biological Systems Section Score Mean = Std. Deviation = Section Percentile Critical Analysisand Reasoning Skills Score Rank Section Score Mean = Std. Deviation = Biological and Biochemical Foundations of Living Systems Percent Biological and Biochemical Foundations of Living Systems Section Score Psychological, Social, and Biological Foundations of Behavior Percent Psychological, Social, and Biological Foundations of Behavior Section Score Section Percentile Biological and Biochemical Score Rank Foundations of Living Systems Section Score Mean = Std. Deviation = Section Percentile Psychological, Social, and Biological Score Rank Foundations of Behavior Section Score Mean = Std. Deviation =

23 Using Confidence Bands in 2016 Admissions Like scores from other standardized exams, MCAT scores are not perfectly precise. The new MCAT score report includes confidence bands that describe the precision of applicants total and section scores and show the ranges in which examinees true scores probably lie. Score reports show confidence bands both numerically and graphically. MCAT total scores are reported with a confidence band of plus or minus two points, and MCAT section scores are reported with confidence bands of plus or minus one point. The example below shows a total score of 504 and its confidence band, which ranges from 502 to 506. The diamond shape shows the confidence band graphically and indicates that the best estimate of an applicant s true score is in the center of the diamond, where the diamond is the tallest and the shading is the darkest. Score Confidence Band Percentile Rank of Score MCAT Total Score % Confidence bands remind you not to overemphasize small differences in scores. When comparing two applicants scores, it is important to consider how much their confidence bands overlap. The greater the overlap in confidence bands, the less meaningful are the differences between the scores. This chart below displays four pairs of MCAT total scores with confidence bands that differ in their degree of overlap. The top pair of scores, 504 vs. 503, is only one point apart on the total score scale. The chart shows that the confidence bands for these scores overlap on four score points 502, 503, 504, and 505. In comparison, the bottom pair of scores, 504 vs. 500, is four points apart on the MCAT total score scale and has confidence bands that only overlap on one point on the total score scale 502 indicating that these scores are less comparable to each other than the first pair. Scores that are closer together have confidence bands that overlap. Applicants' Scores 504 v v v v

24 Using Score Profiles in 2016 Admissions Score profiles highlight applicants strengths and weaknesses across the four sections of the exam through reported scores for each section. The strengths and weaknesses on the exam can be considered with other information about applicants academic preparation and in relation to your institution s missions and goals. Remember that each of the four sections of the new exam includes more test questions and reports more reliable scores than the old exam. More reliable test scores provide more precise information at the section level and support stronger comparisons across sections. MCAT Score Profile Section Score Profile 1 Chemical and Physical Foundations of Biological Systems Critical Analysis and Reasoning Skills Biological and Biochemical Foundations of Living Systems Psychological, Social, and Biological Foundations of Behavior For the four sections, non-overlapping confidence bands show a test taker s likely strengths and weaknesses. Overlapping confidence bands suggest that there are not meaningful differences in performance between sections. 20

25 Making Decisions about Applicants in the 2016 Admissions Cycle In the 2016 admissions cycle, we expect that about half of the applicants will submit scores from the new exam, and about half will report scores from the old exam. Your applicants likely were encouraged to take the MCAT exam when they were best prepared to test. So, for some, that meant taking the old exam, and for others, it meant taking the new exam Applicant Pool examinee with = scores from old exam examinee with = scores from both exams examinee with = scores from new exam Evaluate applicants with scores from the old exam the way you always have. Evaluate applicants with scores from both exams using new scores and their percentile ranks. Evaluate applicants with scores from the new exam using percentile ranks. Designing Thresholds for Screening and Interviewing If you have traditionally used a threshold for deciding which applicants should receive secondary applications or which should be invited to interview, you may want to continue that practice. If MCAT scores are among the criteria you use to decide who should receive secondary applications or be invited to interview, the percentile ranks for the old exam provide a good starting point for setting thresholds on the new exam, but more thought is needed. Applicants reporting scores from the new exam prepared in biochemistry, psychology, and sociology and practiced using the scientific inquiry and reasoning skills tested on the new exam. The old percentile ranks do not reflect that preparation. Especially in this first year, you are encouraged to use your previous thresholds as a starting point, and then lower them a little so you don t miss applicants who might be good candidates for your school. For example, if in the past, you invited applicants to complete secondary applications (or to interview) if they scored at the 60th percentile on the old exam, you might want to look at applicants who score at the 50th percentile on the new MCAT exam. If the course taking, grades, and other information about the academic preparation of these applicants look good to you, this lower percentile rank might be a good threshold for your 2016 screens using the new exam. Looking at applications below your historic thresholds may help you find candidates who meet your mission and goals. 21