Practice Trends and Considerations

Transcription

1 Test Development Practice Trends and Considerations The First in a Series of Peer-Group Reports 3rd Quarter 2011 Introduction Modern societies rely on the development of numerous specialized professions in order to ensure the proper and safe operation of critical societal functions, and the continuing discovery of new knowledge for professional advancement. Fundamental to all professions is some testing/assessment technique or mechanism for admitting new practitioners and for making sure existing practitioners are up to date on their knowledge/skills levels. This Prometric-sponsored report presents data from a nationwide Prometric commissioned a survey of test development professionals responsible for managing their organization s certification process to understand how professional testing and certification organizations meet their needs and the areas where they experience both success and difficulty. Telephone interviews were conducted with 82 individuals between February 25 and March 18, The survey instrument included a mix of constrained response and open-ended questions. survey of professionals directly involved in or managing test content development. The data presented here show test development processes that rely considerably on informal, improvised mechanisms and practices, particularly for item writing and job analysis. Paradoxically, these are areas of the test development process that professional organizations tend to manage in-house, rather than utilize external resources, according to the Prometric survey. Additionally, the data show that relatively few testing and certification organizations have taken advantage of platform-neutral, industry-standard test formats that facilitate collaboration in item Number of Employees at Respondent Organizations (%) No. Employees Percentage of Respondents Less than 20 32% % % 300 or more 20% development and exchange of test material and data across systems. Background to the Survey Telephone interviews were conducted with 82 individuals between February 25 and March 18, The survey questionnaire included a mix of constrained response and open-ended questions. Respondents represented organizations of varying size and from a broad range of industries and professions, including accounting, construction, education, healthcare, information technology, law and telecommunications. Survey Respondent Profile Industry No. of Respondents Accounting and Finance 4 Building and Construction 3 Consulting 5 Education 7 Government and Policy 6 Healthcare 20 Information Technology 8 Industry No. of Respondents Legal 1 Manufacturing 1 Non-profit (unspecified) 1 Professional Association 4 Science 2 Telecommunications 2 Testing and Certification 18

2 This report provides an overview of survey responses related to five critical areas of the test development process: 1. Item Development 2. Test Design and Construction 3. Job Analysis 4. Item Analysis and Psychometrics 5. Test Publishing Subsequent reports provide closer examination of each of these components of developing and delivering effective professional credentialing exams, with specific focus on areas that are particularly challenging to organizations that offer credentialing programs. Current Practices Examination of current practices of testing and certification organizations provides a useful window for observing the existing state of the exam development process. Because exam development requires highly refined processes with ongoing oversight and management, a key question is whether organizations tend to in-source work in the five critical areas or whether they use outside resources. Core Requirements Of the organizations surveyed, many believe that creating exams that are valid and reliable requires ongoing practices that involve a lot of effort and discipline. Processes that are well designed and managed must be in place for the development and delivery of high-quality, sound, valid, reliable and legally defensible exams: 1. Internal Validity and Intelligibility exam items (especially constrained response questions) have one correct answer and are comprehensible 2. External Validity results from exam items and exams match various levels of preparedness of examinees; individual items are predictive of overall performance on the exam 3. Construct Validity exam items and the exam as a whole accurately and adequately capture the activities and required knowledge base of a given profession or activity 4. Reliability test results are consistent across time, individuals and settings In theory, in-house management and development allows an organization to leverage its subject matter expertise to greatest effect, particularly for item development, test construction and job analysis. Conversely, using outside vendors specializing in test development, item analysis and publication leverages their expertise in managing the overall process and carrying out functions that require sophisticated techniques for ensuring the validity, reliability and security of the exam. In practice, according to the Prometric survey, this generally holds true. Table 1 shows that organizations are more likely to in-source test development (item development, test design, job analysis), and use external resources for other specialized aspects of testing (item analysis and test publishing). The majority of respondents said they performed test design (52%) and job analysis (57%) using in-house resources exclusively, while 45% of respondents said they used in-house resources exclusively for item development. Table 1 In-house or Outside: Five Critical Areas of Test Development Resource Item Test Item Job Test management development design analysis analysis publishing In-house 45% 52% 38% 57% 37% Outside 13% 19% 28% 13% 41% Both 43% 30% 33% 30% 22% 2

3 Still as a group, the 75 organizations surveyed demonstrate wide variability in their approaches to the test development process. Additionally, many of the respondent organizations employ mixed approaches, often using both internal and external resources and personnel at all stages of development. They also often use improvisational and informal methods for creating exam items, designing tests, and compensating item and test developers. Item Development: Writing and Assistance The vast majority of respondents (95%) said their test development process is documented, as illustrated in Table 2. Though certainly a good practice, documenting processes can as demonstrated by survey responses involve widely varying levels of detail and completeness of coverage. Also, the documented processes can vary widely in terms of their robustness for the requirements of the task. Recruitment of Item Writers and Subject Matter Experts (SMEs) Table 2 Is your test development process documented? Yes 95% No 4% Don t Know 1% The fundamental elements of all exams are the individual items. Notably, respondents as a group indicated considerable flexibility, informality and variability in approach for item writer recruitment and management. Table 3 Which of the following types of professionals do you use when you write test items and questions? (Select all that apply) In-house SME 39% Industry members of the association 60% Outside experts 56% Others 0% Don t know 5% When asked which types of professionals they use to write the test items, 60% percent said industry members within their organization depicted, as in Table 3. No doubt these individuals are highly qualified in their field of expertise. Writing exam questions, however, requires additional skills, training, experience and time, which often eludes the occasional item writer. Table 4 How many types of professionals (in-house, association members, outside experts) do you use when you write test items and questions? Experts from 1 source 40% Experts from 2 sources 43% Experts from 3 sources 17% Table 4, on the other hand, shows that a strong majority (60%) of respondents said their organizations used experts from more than one source. Most used experts from two sources (43%), and a substantial minority used all three sources in-house staff, associated members and outside experts. 3

4 Drawing on subject matter expertise from several sources could serve as a critical tool for ensuring that questions address the most up-to-date knowledge and practices in a particular field. Responses, however, to two other questions support for test writers and compensation of SMEs suggest a more improvised approach to the demanding task of writing test items and designing exams. Writer Assistance Survey respondents mentioned a number of ways in which their organizations assist item writers in meeting the objectives of their assignments. The three most common types of assistance to item writers were: (1) training, mentioned by 41% of respondents; (2) meetings or workshops, mentioned by 18% of respondents; and (3) feedback, mentioned by 16% of respondents. Because some organizations offered both training and meetings/workshops, the total percentage of organizations that offered at least one of these forms of direct instruction was 50%. Small percentages of respondents also mentioned reference materials, support or mentoring, guidelines, infrastructure (usually referring to online tools), timelines, examples and working as a team. Additionally, 15% of respondents said that no assistance was provided or that they did not know what or if assistance was provided. How do you assist item writers in meeting objectives? Training Meeting/ Workshop Feedback Reference Material Support/ Mentoring Guidelines Infrastructure Timeline 7% 7% 6% 11% 10% 18% 16% 41% Some respondents reported using a more hands-on process with direct access to test development staff and electronic tools: During workshop we have staff and item writers, and they are given a computer and access to item writing software. Staff reviews and coaches them. Examples Work as a Team 5% 4% 0% Another respondent emphasized more of a collegial, if informal, process: Quite simple we provide them with a basic framework and set of objectives. Give them lunch, pay their expenses and keep them comfortable. Make it fun. Other respondents reported having little interaction, training, coaching or feedback: We rely on the chairman sending reminders. We drop [a writer] if not able to keep up. Another said: We provide basic training. Then mainly, it s learning as you go. Job Analysis: SME Compensation The way organizations compensate their SMEs also varies widely, and often involves informal means of compensation like offering honoraria, stipends, gifts and education credits or certification. The most common approach is not compensating at all, with 33% of respondent organizations relying on volunteer labor for at least some of their subject matter expertise. Almost 1 in 4 respondents (20 of 82) said they relied exclusively on unpaid SMEs. Another 11% relied exclusively on in-kind rewards, mostly in the form of education credits and recertification. In all, a majority of respondents (57%) said their organizations paid either nothing or offered small honoraria or in-kind compensation to some or all of their SMEs. 4

5 In all, a majority of respondents (57%) said their organizations paid either nothing or offered small honoraria or in-kind compensation to some or all of their SMEs. Here again, the way SMEs are compensated reflects broader test development practices that generally rely on informal processes and mechanisms for obtaining professional input. The professionals who volunteer or accept small rewards for their time and expertise are likely dedicated and successful practitioners, committed to promoting the integrity and advancement of their field. Still, processes that rely on mixed, ad hoc approaches reliant on the volunteer contributions of busy professionals are more likely to be fraught with inefficiency and less than optimal output. How do you compensate your subject matter experts? Select all that apply. Not Compensated Per Diem Part of Salary In-kind Per Item Written Item Analysis and Psychometrics Honorarium/ Item analysis and psychometrics are two areas of test development and Stipend 9% deployment that are most likely to be completed or assisted by third-party vendors. In general terms, item analysis ensures that Per Item Used 5% performance on individual test items are good predictors of performance Other 11% on the test overall, and that distractors (incorrect responses in a multiple choice question) are attractive to examinees who do not 0% know the correct answer. Psychometrics goes further by analyzing the knowledge, skills, attitudes and personal traits of test takers, and the impact of these factors on test results. 13% 16% 15% 25% 33% Because item analysis and psychometrics require highly specialized analytical skills, it makes sense that testing organizations are more likely to use outside resources to assist with this task. It is important, however, for item writers and test designers to understand the role and use of psychometrics in the test development process in order to enhance the quality of test items and improve writing efficiency. Table 5 Do you provide training to content experts regarding the role and use of psychometrics in your organization s test development process? Some 61% None 30% Don t know 9% A majority of respondents (61%) reported that they provide at least some training to content experts regarding the role and use of psychometrics in the test development process. Almost one-third (30%) said they provided no such training, while another 9% did not know. Among those organizations that did provide some training, the amount and specificity of training varied from short overviews to more in-depth training. One respondent said that training is provided in written background materials at a meeting to give an overview. Other respondents mentioned giving a one-hour presentation on the use of psychometrics. 5

6 Many respondents made general statements that such training is provided, while others were quite detailed in the types of analysis that are presented to their item writers. Test Publishing: Industry Standard and Proprietary Formats Test publishing is the process by which the structure of the exam is developed, not the just the actual delivery of the exam. As we saw in Table 1, 63% of respondent organizations use third-party vendors for all or part of the test publishing process, with the remaining 37% completing this task exclusively in-house. In terms of test format, only 22% of the respondents said their organization used industry standard formats like the Question and Test Interoperability (QTI) specification. This is notable because the cost in terms of time and labor needed to develop and validate good questions can be reduced considerably by using a format that is platform and technology neutral. Table 6 What formats does your organization use to publish exams? Industry standard format (e.g., QTI) 22% Vendor specific proprietary format 50% Other format 17% Don t know 11% As Table 6 shows, half of the respondents (50%) said they used a vendor-specific proprietary format. Another 17% said they used some other format. For this last group, formats included using simple word processing formats like Rich Text Format (RTF) and general design software like Adobe InDesign. The low rate of adoption of interoperable, standard test publishing formats suggests that many organizations involved in the certification process have so far not taken advantage of the considerable efficiencies that these standardized formats offer for exam creation, item analysis, publishing, scoring and reporting. Question and Test Interoperability (QTI) QTI was developed by the IMS Global Learning Consortium, a non-profit consortium of industry and academic institutions dedicated to developing interoperability standards and adoption of practice standards for distributed learning. The QTI specification defines a standard format for representing exam content and results. Standard formats support the exchange of test material and data between test authoring and delivery systems and allow exams to be delivered on multiple systems. QTI consists of a data model that defines the structure of questions, exams and results. Conclusion The survey data presented here show varied approaches to the test development process. Many of these approaches involve informal, improvised mechanisms and practices. This is especially true for the item writing and job analysis stages but can also be seen in item/test analysis and test publication. Responses to the survey indicate there is considerable opportunity among organizations who offer credentialing exams to enhance quality and increase efficiency in their test development processes. This is the first in a series of peer-group reports based on the nationwide 2011 Prometric Survey on Test Development. It provides an overview of current practices in five critical areas of test development: item writing, test design, job analysis, item analysis and test publication. Subsequent reports will provide a more in-depth presentation of the issues and challenges that testing and certification organizations confront in these five areas. 6

7 Prometric, a wholly-owned subsidiary of ETS, is a trusted provider of technology-enabled testing and assessment. Its market-leading test development and delivery solutions allow clients to develop and launch global testing programs as well as accurately measure program results and data. Prometric reliably delivers and administers more than 10 million tests a year on behalf of approximately 400 clients in the academic, financial, government, healthcare, professional, corporate and information technology Find Out More Learn more about our Test Development Solutions by visiting or by calling toll-free markets. It delivers tests flexibly via the Web or by utilizing a robust network of more than 10,000 test centers in more than 160 countries. Copyright 2011 Prometric, Inc. All Rights Reserved.