Testing Integrity Practices and Procedures for Online and

Size: px

Start display at page:

Download "Testing Integrity Practices and Procedures for Online and"

Corey Waters
8 years ago
Views:

1 Testing Integrity Practices and Procedures for Online and Computer based Assessments Wayne J. Camara The College Board NCES Sponsored Symposium ontesting Integrity 2/28/2012

2 CBT vs Paper Online testing offers numerous advantages over P&P testing, ti including features which h can improve test t security. As with all assessments, the intended purpose and potential consequences is suggestive of the types of threats to test integrity we need to focus upon. Threats to all assessments: item exposure, candidate authenticity, data transmission & storage, proctor and personnel integrity, system integrity (prevent interruptions and irregularities)

As with all assessments, the intended purpose and potential consequences is suggestive of the types of threats to

3 Assessment Purposes and Threats to Testing Integrity Cheating increases with age of student, bandwidth & distance (Rowe, 2004). Summative assessments different threats emerge for different intended uses of scores: 1. School and district accountability 2. Student rewards ds(endorsed diploma, dpo a,entry into college credit bearing course) 3. Teacher and educator accountability (financial incentives or penalties, disciplinary based actions) 4. Student barriers (graduation, retention, mandatory developmental programs, college remediation courses)

Student rewards ds(endorsed diploma, dpo a,entry into college credit bearing course) 3.

4 Testing Integrity: Unique risks with CBT Student High Stakes Teacher/Educ. Account. Extended testing window Students disclose items/tasks to other students or post on social networks to those testing later in window Educators provide instruction on specific tasks to aid students testing later in the window Performance Easier to recall, more difficult Provide procedural solutions. tasks to create comparable tasks Instruction targeted to tasks Reuse of items exposure rate Testing environ CBT lab Greater chance of intentional or nonintentional disclosure Easier to hide prohibited materials behind screen. Use handheld devicesto cheat. If reused over years (pre testing, equating) greater risks of teaching to task / item Teacher monitors and changes student responses. Small groups settings present greater challenges Assisting Same risk as P&P unless items Teacher views student progress and students during spiraled. Need privacy carrels responses may offer hints to testing individual student or group Technology Machine allows access to external web resources? NA

testing later in the window Performance Easier to recall, more difficult Provide procedural solutions.

5 Processes/policies that could mitigate risks to integrity of CBT test results Processes and policies must be tailored to the types of risks or threats to test integrity that are anticipated based on the intended use, stakes and consequences for school, students and educators. Reduce risk of item exposure via extended testing windows with sameform present the biggest security threat when tests used for high stakes: More robust item banks and spiraling Use of multistage adaptive models Linear forms require more forms for the same testing window or single use Reuse of items operationally, for equating or pretesting: Reuse of scenarios, simulations, or extended performance tasks can more easily be captured and hence have less validity when exposed for any length of time. Limit disclosure and reuse over several years. Limit reuse of performance tasks (extended multi year window w/out release or develop hundreds of tasks to pool from) Limit retesting different forms/item pools.

Reduce risk of item exposure via extended testing windows with sameform present the biggest security threat when tests used for high stakes: More robust item banks and spiraling Use of multistage

6 Recommendations: Processes/policies that could mitigate risks to integrity of CBT test results Administration and Scoring Reduce opportunity for cheating send message cheating is not tolerated. Classroom teachers should not be administering tests to students in their classes there is simply too much temptation Proctors should have no stake in outcome or risk collusion. Environment should preclude copying responses from students seated adjacent (spiraling, different forms, or some physical obstruction); Document seating and proctors. Mandatory training of proctorsand administrators handling test materials ; verify understanding of appropriate test procedures and consequences of unauthorized procedures. Student reads and signs statement like an honor code or integrity ypolicy. Prohibit all handheld electronic devices (smartphones, calculators) Employ variety of item formats & constructed response tasks to reduce ease of cheating. Impose conditions on retest opportunities beware of students unplugging equipment to restart or retest.

Environment should preclude copying responses from students seated adjacent (spiraling, different forms, or some physical obstruction); Document seating and proctors.

7 Recommendations: Processes/policies that could mitigate risks to integrity of CBT test results Technology Prepare for unexpected it will occur Ensure students can notaccess web resources (outside the system) Items and data are encrypted and stored on secure server (not desktops). Paper forms use different item banks and chain of custody established. Audit social networks, school preparation, blogs. Ensure high system reliability outages, interruptions and irregularities which require candidates to stop and start, retest, or complete paper forms. Guard against sniffers that decipher and read items/responses and attempts to have test administrators disclose passwords (McClure et al, 2001). Disable network capabilities, printers. Conduct tformal web crawling bf before/after. /ft Use Intrusion detection software to catch attacks prior to their occurrence. Backup grade book or roster in case of attack and chances. Statistical Checks on aberrance rates, retest or score volatility statistics (individual, site) does data conform to test response models? Check on irregular latencies, response patterns. High/Low Aberrance score, Cheating index, Thresholds (Impara et al, 2005) Distance assessments When online performance exceeds traditional tests Have some traditional assessments (Rowe, 2004).

Audit social networks, school preparation, blogs. Ensure high system reliability outages, interruptions and irregularities which require candidates to stop and start, retest, or complete paper forms.

8 National Council for Measurement In Education (NCME) Draft Guidelines on Testing &Data Integrity Data integrity is shared ethnical and professional requirement. Need to develop and implement a comprehensive data integrity policy and why its important. Til Tailored dto use of test. t Training for all levels with examples of unacceptable behaviors (nondisclosure, confidentiality, participation forms) Proactive prevention eliminate opportunities. Comprehensive data collection and maintenance Comprehensive policies for reporting cheating, security breaches, suspicious activities (db & investigations) Biometrics, data forensics, statistical patterns, etc.

t Training for all levels with examples of unacceptable behaviors (nondisclosure, confidentiality, participation forms) Proactive prevention eliminate opportunities.

plantemoran.com What School Personnel Administrators Need to know

plantemoran.com What School Personnel Administrators Need to know plantemoran.com Data Security and Privacy What School Personnel Administrators Need to know Tomorrow s Headline Let s hope not District posts confidential data online (Tech News, May 18, 2007) In one of