Analysis of Test Item Content for the M edical Assistant Test National Association of Health Professionals August 31st, 2011 1
Introduction In this brief report, an evaluation of the item content of the Medical Assistant Examination of the National Association for Health Professionals is presented. Periodic evaluation of item content helps assure the relevancy of test to current educational curriculum and to job practice. Analysis of the evaluations provides evidence for the content aspect of test validity. Typically, a panel of experts from varying backgrounds, including educators and practitioners, rate the items for relevancy. These ratings, along with qualitative comments, can provide a basis for deleting problematic items, changing the representation of the categories in the test blueprint or introducing new categories. The Medical Assistant test has been used operationally to provide feedback to educators and to certify the proficiency of medical assistants for 30 years. Some items have been retained on the test over the years. It is particularly important to re-evaluate the older items for continued relevancy. Method Items evaluated. Equal numbers of items were selected from each of the eleven categories of the test blueprint. Within categories, the items that have been on the test for the longest time period were selected. Evaluation panel. The evaluation panel consisted of 100 members from diverse backgrounds that are relevant to the job of medical assistant. The positions of the selected panel members is shown on Table 1, along with the percentage of panel members within each position. It can be seen that medical doctors (MD), registered nurses (RN), licensed practical nurses (LPN) and medical assistants are approximately equally represented. Medical instructors were less represented on the panel. Hence, the panel is mainly represents job practice rather than training. Procedure. Each panel member was given all 33 items to be evaluated. Panel members were individually sent the items and instructions. Each item was rated for relevancy to the job of a medical assistant on a three point scale defined as follows: 1, Not relevant, 2, Relevant and 3, Very Relevant. Results Table 2 presents the mean ratings by category, based on the three-point rating scale described above. It can be seen that the overall mean rating (M = 2.7273) is closest to the highest category scale point of 3 for Very Relevant. Overall, items varied only somewhat from the mean rating (SD =.24402). The means of the items with each blueprint category is also shown on Table 2. These means varied little across categories (2.5333 M 2.9333). The category with the lowest mean is Law and Ethics, followed closely by Insurance and Disease. However, the mean for these categories remains closer to the scale point for Very Relevant (3) than for Relevant (2). The categories with the highest means were Medical Terminology and Pharmacology. Also shown on Table 2 is the number of items with each category on the Medical Assistant Test. The lowest rated category, Law and Ethics, is represented by relatively few items on the test. One of the two highest rated categories, Medical Terminology, has the largest number of items on the test. The other highest rated category, however, Pharmacology, has substantially fewer items on the test. 2
Table 3 presents the percentage of raters endorsing each relevancy category for each item. With some exceptions, the majority of items were rated in the Relevant or Very Relevant category. The exceptions are Item 14, Item 3 and Item 9, with 20% of the ratings in the Not Relevant category. Discussion The results are very positive about the relevancy to job practice of the items that were reviewed. The overall mean rating of relevancy was very high, close to the scale point of Very Relevant. Further, items differed little from the overall rating, as the standard deviation was quite small. Finally, only three items had a small percentage (i.e., 20%) of ratings in the lowest category. This finding could be due to the varying background of the panel members. All other ratings were in the Relevant and Very Relevant category. The data does not provide a rationale for deleting any of the items that were evaluated for being not relevant to the job of medical assistant. Further, few differences were observed in the relevancy of the evaluated items for the various categories of the test blueprint. The mean ratings for items in each categories was close to the highest category relevancy. Although the difference in mean relevancy ratings between the categories did not completely correspond to their relative frequency on the test, this effect could be due to the particular items that were evaluated. A full comparison of the relevancy for all items in the categories, and perhaps global ratings of the categories, could further evaluate category representation. Table 1. Background of the rating committee. Profession Percentage MD 25% RN 25% Medical Assisant 20% LPN 20% Medical Assisant Instructor 10% 3
Table 2. Mean ratings by category. Blueprint Category Mean Number of Items on Test Items Evaluated Std. Deviation Anatomy and Physiology 2.7333 18 3.30551 Body Planes 2.6667 5 3.23094 Disease 2.6000 15 3.20000 Insurance 2.6000 12 3.20000 Laboratory 2.7333 30 3.30551 Law and Ethics 2.5333 7 3.41633 Medical Procedures 2.5333 30 3.23094 Medical Terminology 2.9333 50 3.11547 Nutrition 2.8667 7 3.11547 Office Procedures 2.8667 14 3.11547 Pharmacology 2.9333 12 3.11547 Total 2.7273 33.24402 4
Table 3. Proportion of responses in three rating categories. Blueprint Category Item Not Relevant Relevant Very Relevant Anatomy and Physiology 10.00.60.40 11.00.00 1.00 12.00.20.80 Body Planes 13.00.20.80 14.20.20.60 15.00.20.80 Disease 4.00.40.60 5.00.20.80 6.00.60.40 Insurance 16.00.60.40 17.00.40.60 18.00.20.80 Laboratory 1.00.20.80 2.00.00 1.00 3.20.20.60 Law and Ethics 7.00.00 1.00 8.00.80.20 9.20.20.60 Medical Procedures 19.00.60.40 20.00.20.80 21.00.60.40 Medical Terminology 22.00.20.80 23.00.00 1.00 24.00.00 1.00 Nutrition 25.00.00 1.00 26.00.20.80 27.00.20.80 Office Procedures 28.00.00 1.00 29.00.20.80 30.00.20.80 Pharmacology 31.00.20.80 32.00.00 1.00 33.00.00 1.00 5
6