National Data. Reasons for Consideration. Guidelines or Best Practices. Regulations 3/5/2013

Transcription

1 Use of Standardized Examinations in Nursing Education Programs Finding the Balance in Testing Improving Performance and Setting Consequences Virginia D. Ayars, EdD, MS, Texas Board of Nursing Austin, TX Lisette Barton, PhD,, FNP-BC School of Nursing, University of Houston-Victoria, Sugar Land, TX Presenter: Virginia D. Ayars, EdD, MS, Nursing Education Consultant Texas Board of Nursing March 2013 Reasons for Consideration Emotional issue for students and family Reports to Boards of Nursing Attorneys hired Legislators contacted National Data Survey completed in February 2013 Administered through NCSBN web survey tool Education consultants from 35 of 60 member boards participated The NCSBN membership comprises boards of nursing in: Fifty U.S. states District of Columbia Four U.S. territories-american Samoa, Guam, rthern Mariana Islands, and the Virgin Islands Four states have two boards of nursing, one for registered nurses (s) and one for licensed practical/vocational nurses (LPN/s): California, Georgia, Louisiana and West Virginia One state, Nebraska, has both the board of nursing and the board for advanced practice nurses (APs) represented Regulations Does your Board of Nursing (BON) have regulations about the use of standardized examinations in nursing education programs? Guidelines or Best Practices Does your BON offer guidelines or best practices to assist nursing education programs in the utilization of standardized examinations? 1

2 Element of Course Grades Does your BON have a position statement about nursing education programs utilizing standardized examination scores as an element in determining course grades? Student Progression or Graduation Does your BON have a position statement about nursing education programs utilizing standardized examination for student progression or graduation? Benefits What do you perceive to be the benefits to nursing education programs that utilize standardized examinations? Validated psychometric properties Assists in preparing students for NCLEX Increases faculty accountability Assists in program evaluation as compares student performance to national standards Effective remediation tool Challenges What do you perceive to be the challenges to nursing education programs that utilize standardized examinations? If used as part of grade and to prevent progression in program, may not be legally defensible Identification of correct cut-off score Inconsistent grade weights within courses and programs Costs to program or student Programs should rely on sound curriculum and teaching strategies rather than on the standardized examinations National Data Additional Comments: Recommend that programs utilize standardized examinations for the intended purpose Programs need to utilize the exams as a resource to improve program of study and faculty performance rather than as high-stakes exit examinations Ensure that program policies reflect the intended use of examinations to students Texas Board of Nursing (BON) Texas BON does not have purview over the use of standardized examinations Texas BON offers Education Guideline: Using Standardized Examinations BON staff interviewed three vendors to determine the intended purposes in developing the Guideline 2

3 Suggested Purposes An assessment tool of potential or enrolled students An assessment tool of the program s effectiveness An assistive tool to familiarize students with computerized testing Survey Completed in February of 111 Texas BON approved professional nursing education programs deans or directors responded 74 of 99 Texas BON approved vocational nursing education programs directors or coordinators responded Use in Nursing Programs Identification of Exams Utilized 6.8% 93.2% 1.2% 98.8% % 65.4% 61.8% % 1.5% 1 HESI ATI NLN Other (please 38.5% 1.3% 9.0% HESI ATI NLN Other (please Purposes Respondents: Assessment of content mastery NCLEX-PN predictor Assessment of faculty/program effectiveness Respondents: NCLEX- predictor Assessment of content mastery Assessment of faculty/program effectiveness Admission criteria (HESI A2) Frequency of Use During Program % % % % 1 5.0% 33.3% Each course 43.9% Each semester 16.7% Beginning of program 45.5% End of program 21.2% Other (please % 41.3% % 2 1 Each course Each semester Beginning of program 57.3% End of program 26.7% Other (please 3

4 Benefits of Use Incorporate into Course Grade 33.8% 66.2% 13.7% 86.3% % 72.6% 79.0% 77.4% Data to compare to benchmarks Indication of needs of individual students for remediation Indication of curriculum weaknesses/strengths Indication of instructional weaknesses/strengths 17.7% Other (please % 87.5% 86.1% 75.0% Data to compare to benchmarks Indication of needs of individual students for remediation Indication of curriculum weaknesses/strengths Indication of instructional weaknesses/strengths 12.5% Other (please Use of Supplemental Materials % 41.8% 27.3% 214.5% % % 28.8% 13.6% 8.5% Supplemental Materials Respondents: Videos Practice exams Tutorials Respondents: Practice exams Review books Case studies Affect Student Ability to Progress or Graduate 40.3 % Number of Times Student May Retake Exam 25.8% 30.6% % 15.3% 16.7% % 52.1% 47.9% 9.7% 19.4% 14.5% Other (please 9.7% 12.5% Other (please 4

5 Faculty Members Take Exams Exams Reflect Required Content 1.4% 36.7% 63.3% 38.6% 61.4% % 98.6 % Identification of Other Products Respondents: Modules to assist ESL students Medical Terminology modules Increased information re. missed items More case studies, videos/scenarios & practice questions Respondents: More computer adaptive tests Faculty training tools More case studies for simulation lab use Exit HESI linked to other products (case studies, critical thinking exams, testing-writing tools, faculty training) Student Comments Respondents: Hard, but empowering Anxiety producing Hate them, but love them after pass NCLEX! Respondents: Students appreciate value after taking NCLEX Exams are accurate predictors of NCLEX success Exit HESI most helpful to students Offers practice with NCLEX style questions Faculty Member Comments Respondents: Highly positive responses Assist in self-evaluation & course evaluation Validated curriculum content Respondents: Evaluation of faculty performance re. instructional methodologies Helps when counseling students and assists in guiding students to address identified weaknesses Prepares students and validates performance Additional Comments Respondents: Very comprehensive Highly beneficial Concerned about rising costs Respondents: Critical that students clearly understand requirements upon admission to program Need to keep use in perspective Strive for balance between not teaching to the test, yet ensuring students prepared for NCLEX 5

6 Recommendations Placement across the curriculum with planned remediation Academic progression policy to clarify use and frequency of standardized examinations and test results Review of plan by school s legal counsel Academic progression policy published in accessible and public format Changes in policies implemented when student first enrolls in program or at beginning of each class or academic term Ask student to sign receipt of academic progression policy Cautions t recommended to use as High-Stakes Testing Place at appropriate points consistent with didactic content and clinical learning experiences Do not pattern curriculum changes solely upon testing results Be aware of any security issues about the standardized examination test pool Questions? For more information, please contact: Virginia Ayars Virginia.ayars@bon.texas.gov Thank you! Standardized Predictive Testing: Practices, Policies, and Outcomes Lisette Barton, PhD,, FNP-BC School of Nursing, University of Houston-Victoria, Sugar Land, TX Rae Langford, EdD, College of Nursing, Texas Woman s University, Houston, TX Pamela Willson, PhD,, FNP-BC, CNE Past Director of Research, Elsevier Review & Testing, Houston, TX Background Nurse educators use standardized testing, curricular standards and guidelines to direct educational practice. Standardized testing predicts student success and identifies student s weak content areas to allow for a focused remediation plan. Multiple remediation strategies are often implemented simultaneously. Little is known about the effectiveness of preparation, remediation, and retesting policies in nursing educational programs in regards to standardized outcome testing. Purpose The four primary aims of this study were: Describe the predictive accuracy of the HESI Exit Exam for NCLEX- success. Describe schools remediation policies and practices. Determine if students who attend nursing programs with certain high stakes testing policy components score better on the HESI Exit Exam than students who attend nursing programs without certain high stakes testing policy components. Identify which combination of policy components result in higher HESI Exit Scores. 6

7 Conceptual Framework Classical test theory Reliability and Validity Crocker and Algina (1986) Vygotsky s Zone of Proximal Development Third Voice Developmental goal attainment Eun, Knoetek, & Heining-Boynton (2007) A random stratified sample was drawn from Elsevier s HESI Exit Exam users 1 September 2009 through 31 August 2010 SON Deans/Directors were queried via password protected survey 471 schools were invited; 99 schools responded for a 21% response rate Phase I Testing Policies & Practices Of the 99 schools, 64 reported testing policies and practices for a 65% response rate Phase II Predictive Accuracy of HESI Exit Exam Of the 99 schools, 69 reported NCLEX- outcomes for a 70% response rate Phase I and Phase II Invited Respon ded Total AD BD Response Rate Schools % Students Phase I Instrument HESI E2 Exit Exam Practices/Strategies Questionnaire 29-item multiple choice and fill in the blank electronic survey Exit Exam program policy Benchmark use Strategies used to prepare for the Exit Exam Duration of time allotted to preparation Re-testing practices Remediation required between testing Consequences of the Exit Exam Data HESI Exit Exams Scores 160-item (10 pilot items) comprehensive exam Most schools administered the E 2 within 6 weeks prior to completion of the nursing curriculum NCLEX- Outcomes First time test takers Pass/Fail; Unknown; Did not graduate HESI E2 Exam Practices/Strategies Questionnaire Preparation Components Remediation Components Consequence Components Phase I Results: Testing Policies and Practices (n=64) Descriptive Data Required Preparation Plan Testing a Component of Course Grade Mandatory Remediation Remediation Type Number of attempts Consequences Correlation Data Testing Policies and HESI E2 Score Remediation Strategies and HESI E2 Final Score 7

8 Preparation Plan 39 of 64 schools provided students with an exit exam preparation plan (61%) 33 schools required students to use the preparation plan (85%) School s timeframe 32 (82%) 6 weeks ahead 30 (77%) more than a week ahead Preparation Plan (61%) Types of Preparation (Multiple components used) Self-guided review (eg., case studies, study guide, NCLEX type items) Frequency Percent (N=39) Faculty guided group review Faculty guided individual review Formal review (eg., HESI, ATI, Kaplan) Peer/mentor tutoring Self-guided formal review course (online) 5 13 Exit Exam as Course Requirement 46 schools (67%) used the E2 as a course requirement 11 schools exam for course completion 11 schools pass/fail criteria 11 schools counted exam as 5-10% of course grade 10 schools counted exam as 20-30% of course grade 3 schools counted exam as greater than 30% of course grade Phase I Descriptive Results Mandatory Remediation 41 of 64 (64%) schools required remediation after the initial failure to meet benchmark 15 (37%) schools required students to pay additional fees for remediation 37 (90%) schools required proof of participation in remediation Benchmark Policy Mandatory Benchmark Mandatory Re-Testing Benchmark & Re-Testing N = 64 Schools 54 schools (84.4%) had an exam policy 37 (69%) used 850; 10 (19%) used 900 Range was schools (38%) of the 64 schools achievement of benchmark was mandatory 37 schools (58%) required re-testing 1 time 15 schools; 2 times 9 schools; 3 times 12 schools; 4 times 1 school Consequences Of the 54 schools that had a benchmark: 30 schools (56%)set consequences for NOT meeting the benchmark Top 3 1 st Course failure (15) 50% 2 nd Delay NCLEX candidacy (13)24% 3 rd Delay graduation (9)17% Multiple consequences Repeat applicable course Audit applicable course Delay/Deny graduation Delay/Deny NCLEX candidacy Program removal 8

9 Phase I Correlation Results: Testing Policy and Practices with HESI E2 Score Policy Significance Achieving the HESI E2 benchmark score is mandatory Require students to participate in preparation plan Require students to retest Require remediation after a student fails to meet the benchmark n = 1426 Mean = SD = n = 1469 Mean = SD = n = 2429 Mean = SD = n = 2618 Mean = SD = n = 2908 Mean = SD = n = 3144 Mean = SD = n = 2184 Mean = SD = n = 1995 Mean = SD = p <.0001** p <.0001** p <.0001** p <.0001** Phase I: Remediation Policy Strategies: Correlations to Last HESI E2 Score (N=5038) Correlation P value Formal Review Course ** Tailored Self-Guided HESI Exam Review Self-Guided Online Review course ** ** Self-Guided Case Studies ** Peer/Mentor Tutoring ** Phase II Results: Predictive Accuracy of the HESI Exit Exam for NCLEX Outcomes (N = 5038) HESI Exit Exam Category 900 and above Student NCLEX Outcome Passed Failed Predicted Accuracy or less TOTAL Conclusions The HESI E2 Comprehensive Exam is predictive of NCLEX success. Schools of Nursing are designing and implementing high stakes testing policies to manage the implementation of the HESI E2 Exam. Several components within a high stakes testing policy are significantly related to individual scores on the HESI E2 Exam. Certain types of remediation policies can be correlated with HESI E2 individual scores. High stakes testing policies need to be crafted to include certain components that are related to HESI E2 success. 9