Passing Scores: Background and Issues

Transcription

1 Passing Scores: Background and Issues Sharon E. Osborn Popp, Ph.D. Testing Specialist/Psychometrician, WREB July 25, 2012 Overview of Passing Scores in Licensure. The Standards for Educational and Psychological Testing (AERA, APA, NCME; 1999) characterize the purpose of tests that serve the needs of professional and occupational credentialing as identifying practitioners that have met a set of standards. The standards are high, in order to protect the public and the profession by excluding unqualified individuals, but not so high as to unduly restrain the right of qualified individuals to offer their services to the public (p.156). The process of setting the passing standard must, therefore, be credible, legally defensible, and well-informed, in order to protect the public as well as the rights of candidates. Several issues related to passing scores and setting passing scores are summarized here, including legal defensibility and the validity of passing scores. Absolute (Criterion-referenced) versus Relative (Norm-referenced) Standards. There are two basic types of passing standards for assessments: relative, or norm-referenced, and absolute, or criterion-referenced. Relative standards compare examinee performance to others taking the test, e.g., a standard that states only the top 10% shall be admitted into a program. Relative standards are not appropriate for professional licensure, because the standard varies with the ability of the sample taking the test. With a relative standard, some examinees will pass and some will fail regardless of their preparedness and knowledge. Further, according to Norcini (1994), "relative standards may be interpreted as a manipulation of manpower rather than a determination of competence. This could raise restraint of trade issues for most licensing and certifying bodies in the United States" (p.166). Standard in the Standards for Testing states that the "level of performance required for passing a credentialing test should depend on the knowledge and skills necessary for acceptable performance in the occupation or profession and should not be adjusted to regulate the number or proportion of persons passing the test" (p. 162; AERA, APA, NCME; 1999). The passing standards set by WREB examination committees are absolute, or criterion-referenced. An absolute standard is set to reflect a standard of knowledge and practice. Theoretically, all candidates could pass or all could fail when compared to an absolute standard. In practice, pass rates of 100% and 0% are unlikely when a credible and defensible passing standard has been set. For many credentialing examinations, the vast majority of candidates are very well-prepared, so relatively high pass rates are not unusual. Raising a passing standard that has been established within the absolute, criterion-referenced framework to make it more difficult for candidates to pass is akin to imposing a relative standard. Legal Defensibility of Passing Scores. Examinations that are used to support a governmental decision to either grant the right to practice a profession or deny that right, are subject to a very high degree of legal scrutiny, which includes investigating the process for determining the passing score. Atkinson (2012) states that "Pass/Fail determinations distinguish between

2 2 candidates who exhibit minimum competence to practice the profession and those who do not exhibit such competence level. In licensure, the obligation on the part of the governmental agency is to assess minimum competence" (p. 508). He explains that certification of advanced competence is generally assessed through the private sector is not subject to the full extent of legal challenge, unlike licensure examinations, where the stakes are very high, i.e., the granting or denial of the right to practice. From a legal perspective, higher scores on a licensure examination do not reflect enhanced qualifications when the passing standard is developed to assess minimum, entry-level competence, consistent with statutory public protection obligations (p. 513; Atkinson, 2012). The Standards for Testing point out that tests used for credentialing need to be "precise in the vicinity of the passing, or cut, score" and may even be designed such that they are precise only in the vicinity of the cut score because they are "not designed to indicate how badly the candidate failed, or how well the candidate passed" (p. 157; AERA, APA, NCME; 1999). Kane, Crooks, and Cohen (1999) explain that the passing standard on credentialing tests must reflect mastery of the knowledge, skills, and judgments that are "critical for practice, in the sense that they are necessary, although perhaps not sufficient, for effective performance in practice," since a candidate also requires other characteristics to practice effectively, such as motivation and ethics (p. 197). The passing standard or cut-off score must balance the need to identify candidates that lack the adequate knowledge, skills, and judgments required to practice safely with the need to identify candidates that are ready to begin practice, even though they may not yet be prepared to manage all possible professional encounters. The Standard-Setting Process - an Example: Setting Passing Scores for the WREB Dental Hygiene Examination. The purpose of assessments that support licensure is to draw inferences about the candidate's readiness for practice. Two major approaches are employed. One approach involves directly observing the candidate's performance within an actual or simulated professional encounter. The other approach requires the candidate to demonstrate professional knowledge, skills, and judgments via responding to a series of tasks or questions. The WREB Dental Hygiene Examination utilizes both approaches. The clinical section of the examination involves judging a candidate's clinical competency in providing oral health care to an actual patient. The written section of the examination evaluates the candidate's ability to utilize professional judgment via a series of questions related to a patient case presented within an interactive computer-based environment. To pass the WREB Dental Hygiene examination, the candidate must pass the clinical and written sections of the examination. The passing standards for each section have been set in accordance with guidelines put forth in the Standards for Educational and Psychological Testing (AERA, APA, NCME; 1999). The Standards for Testing are very clear that cut scores must be determined with respect to professional standards of practice and should not vary in reaction to the needs of the community/marketplace or reflect an arbitrary percentage. WREB's examination committees determine cut scores based on professional standards of content and practice, even when arbitrary cut scores have been legislated, such as 75%." Setting a passing score at 75% without

3 3 evidence to support that the level of performance corresponds clearly to minimal competence is not a credible, defensible standard for a credentialing test; 75% of a difficult test is not comparable to 75% of a less challenging test. WREB re-scales examination scores to be interpreted as "75," but the scores reflect the defensible passing standard set by each professional examination committee. On selected-response test forms, this means that the un-scaled raw score at the passing standard may be higher or lower than 75% of the form's questions, depending on the form's level of difficulty. For performance-based tests, the examination committees define levels of performance with respect to critical aspects of clinical practice. The level of performance that reflects minimal competency is scaled to be a 75. Committee Composition. In discussing the validity of setting passing scores for licensure examinations, Norcini (1994) states, "Defensibility will rest on inclusion of a number of experts in the broad content area and those familiar with the examinees and their level of performance." WREB's Dental Hygiene Committee is responsible for setting the passing standard on the clinical section of the examination and the Process of Care Committee is responsible for setting the passing standard on the written section. Both committees consist of six to seven experienced registered dental hygienists, representing various WREB member states. One of the members on each committee must be an active educator. The inclusion of an educator is critical to the standard-setting process because of their familiarity with the candidate population as potential entry-level practitioners (see e.g., Buckendahl and Davis-Becker, 2012). Experienced subject matter experts tend to set standards unrealistically high, so discussing expectations for the competency of emerging practitioners is vital to the standard-setting process (see e.g., Kane, Crooks, and Cohen, 1999). Committee membership rotates regularly to ensure regional diversity in representation. Supporting each committee in the standard-setting process with examination materials, procedural protocols, and statistical analysis, are WREB staff and a professional psychometrician. Clinical Section Passing Score. The clinical section of the examination is a weighted combination of performance in periodontal assessment (i.e., recording probing depths and gingival recession) and calculus removal. The maximum points possible is scaled to be 100, with a passing standard of 75, based on minimally competent performance on an examinerdetermined sample of tooth surfaces from at least one fully treated quadrant. The passing standard reflects an acceptable quantity of competent clinical behaviors after applicable error points and clinical penalties are deducted. Written Section Passing Score. The written section of the examination is a computer-based assessment, in which a patient examination and a patient's records are presented interactively. Selected-response questions are offered in five sections: General Health and Risk Assessment, Periodontal Assessment, Disease Classification, Dental Hygiene Care Planning, and Outcomes Assessment. Performance within each section is weighted proportionally to reflect the content standards that represent current professional practice.

4 4 The standard-setting process for the written examination involves committee judgments of each item on the exam, according to Ebel's method (Ebel, 1972; Zieky, Perie, and Livingston, 2008). Each committee member must assign each test item to a category that reflects degree of professional relevance (e.g., essential) and degree of difficulty (i.e., the estimated probability of correct response by a minimally competent candidate). Estimated probability values are weighted by relevance and applied to the test form to set a raw passing standard. Raw scores are further scaled to equate among test forms of differing difficulty with 75 as the scaled passing score for each form. Background on the Adjustment of Passing Scores. When a passing standard has been set, appropriate evidence for test validity and the validity of the standard-setting process must be collected and recorded to support the credibility and legal defensibility of the standard (see e.g., Hambleton and Pitoniak, 2006; Kane, 1994). Once established, passing scores may eventually undergo review or revalidation, especially after considerable passage of time or if there have been changes in the profession that dictate a change in test content. In such a case, the passing scores may be set in a new standard setting. Changes in test content or test structure require another comprehensive standard setting process (Mattar, Hambleton, Copella, and Finger, 2012). Sometimes, an "adjustment" of a passing score may take place between the standard settingprocess and the application of the passing standard. Adjustments should be well-documented and justified by a rationale, such as adjusting within a specified proportion of the standard error of measurement to accommodate varying margins of error between test forms, if a need is demonstrated empirically (see e.g., Standard 2.15, Standards for Testing. AERA, APA, NCME; 1999). Even minor adjustments to a passing score can have practical significance to examinees, so Cizek and Bunch (2007) have recommended development of more rigorous and systematic methods and procedures for adjusting cut scores. Delegation of Authority. Regulatory boards may be responsible, statutorily, for conducting all decisions relevant to licensure, including the establishment of passing scores. As identified in the preceding review of issues related to the setting of passing scores, the passing score should not be arbitrary and must be established in a valid and legally defensible standard-setting process. Most professions rely upon outside testing organizations for development and administration of examinations, including the determination of passing scores. However, regulatory boards must be wary of blind reliance on an outside vendor, as the board may be subject to allegations of improperly delegating authority with respect to pass/fail decisions (p. 516; Atkinson, 2012). WREB requires member states to be involved in examination administration, development, and review to: (a) ensure that representatives have multiple opportunities to become familiar with examination content and procedures, and (b) provide input through participation on examination review boards and development committees, which may include participation in the setting and review of passing scores. Atkinson (2012) states, "it is essential that licensing boards understand and participate in the examination process in order to be better prepared in the event of a legal challenge" (p. 534).

5 5 Summary. Issues for consideration in the setting of passing scores on licensure examinations include the type and nature of the passing standard, legal defensibility, the validity of the standard-setting process, and the relationship between licensing boards and testing organizations. Type of Passing Standard. An absolute, or criterion-referenced, standard is recommended for licensure examinations, because the passing score should reflect a defined standard of knowledge and practice. Passing should depend on the candidate's level of ability, not on a pre-specified percentage of passing examinees or on the needs of the community or marketplace. Legal Defensibility. To withstand legal challenge, the passing score on a licensure examination must reflect minimal, entry-level competence, consistent with statutory public protection obligations, and be the result of a valid and credible standard-setting process. Validity of the Standard-Setting Process. The process of setting the passing standard for licensure examinations must be well-informed, credible, and legally defensible, in order to protect the public as well as the rights of candidates. When a passing standard has been set, appropriate evidence for test validity and the validity of the standard-setting process must be collected and recorded to support the credibility and defensibility of the standard. WREB currently maintains a system of recruiting highly qualified subject matter experts that represent various member states, including educators, to participate on the examination committees that engage in setting passing scores. WREB also ensures that passing scores are appropriately implemented according to best practices in psychometric methods and in accordance with national testing standards. Delegation of Authority. Most licensing boards recognize and apply results from examination systems developed by organizations with expertise in testing and assessment; it is essential that boards are well-informed and involved in examination development and administration. WREB requires a minimal level of board representation from member states and encourages high levels of member state participation in the examination process. References: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association. Atkinson (2012) Legal issues and considerations for standard setting in professional licensure and certification examinations. In G. J. Cizek (Ed.), Setting Performance Standards: Concepts, Methods, and Innovations (pp ). New York: Routledge.

6 6 Buckendahl, C. W., & Davis-Becker, S. L. (2012). Setting passing standards for credentialing programs. In G. J. Cizek (Ed.), Setting Performance Standards: Concepts, Methods, and Innovations (pp ). New York: Routledge. Cizek, G. J., & Bunch, M. B. (2007). Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. Thousand Oaks, CA: Sage. Ebel, R. L. (1972). Essentials of Educational Measurement (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp ). Westport, CT: American Council on Education, Praeger. Kane, M. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), Kane, M. T., Crooks, T. J., & and Cohen, A. S. (1999). Designing and evaluating standard-setting procedures for licensure and certification tests. Advances in Health Sciences Education, 4: Mattar, J., Hambleton, R. K., Copella, J. M.,& Finger, M. S. (2012). Reviewing or revalidating performance standards on credentialing examinations. In G. J. Cizek (Ed.), Setting Performance Standards: Concepts, Methods, and Innovations (pp ). New York: Routledge. Norcini, J.J. (1994). Research on standards for professional licensure and certification examinations. Evaluation and the Health Professions, 17(2): Zieky, M.J., Perie, M., & Livingston, S. L. (2008). Cutscores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Princeton, NJ: Educational Testing Service.