Takin Care of Mama From Competency Assessment to Competency Improvement

Transcription

1 M A N A G E M E N T Daniel E. Haun, MT(ASCP)H Argie P. Leach, MHS, MT(ASCP)SH Rita Vivero, MBA, CT(ASCP) Takin Care of Mama From Competency Assessment to Competency Improvement ABSTRACT Although many regulations exist regarding competency assessment, few references show how to design a reliable assessment program for the clinical laboratory. Cytopathology reports reveal that a poorly designed competency assessment program fails to detect incompetence, a finding that we confirmed in our laboratory. We show through case studies and the use of tools how we implemented an effective competency assessment program. The tools helped us to focus the program, design measurements of competency, and create interventions that led to improvements in competency. From the Department of Pathology, Medical Center of Louisiana, New Orleans. Reprint requests to Mr Haun, Department of Pathology, 1532 Tulane Avenue, New Orleans, LA 70112; or dhaun@isumc.edu During the eight-month period of January 1996 through August 1996, our laboratory under-reported leukemic blast cells on 7 occasions. Yet, in 1995 and in 1996, each of our 75 technologists passed an annual WBC morphology competency assessment. This evidence of the inadequacy of our assessment method led us to ask ourselves these questions: How competent is our laboratory? Do we feel 100% comfortable with our laboratory results round-the-clock? If the patient were our mother, would we trust our technologists to correctly interpret her peripheral blood smear or Gram-stained smear? The literature offers surprisingly little about the design of competency measurements in the clinical laboratory, and even less about improving competency. Our purpose is to present a set of tools that helped us to design measures of competency, thus leading to better patient outcomes. First Tool: Takin Care of Mama During the summer of 1996, we formed a competency team and charged its members to operate on the premise that each patient was our mother a mind-set designed to set a high standard for the performance of all laboratory tests. In this light, competency assessment became an absolute necessity instead of a regulatory requirement. This caused us to ask ourselves these questions: Did we want a less-than-100%-competent person working on Mama? Did we want a person who takes shortcuts or has inadequate knowledge to do Mama s laboratory work? Implementing this first tool helped us to focus more on our ultimate goal improved patient outcomes. To focus our efforts, we coined things-thatmatter-to-patients to refer to the health status and satisfaction of our patients, while realizing that their outcomes are the direct result of how well many (not just the laboratory s) healthrelated procedures are executed. Berte and Nevalainen 1 discuss the importance of distinguishing between laboratory outcomes and patient outcomes. At the laboratory level, thingsthat-matter-to-patients include the quality, turnaround time, and cost of test results. Of these, quality and timeliness are primary components of internal competency measurements. In short, our goal was to do the right thing at the right time

2 Second Tool: What Questions to Ask Our next tool involves asking questions about our performance, both as individuals and as a department. Some examples follow: Design the measurement system Are we competent? Adjust Can each technologist calculate body fluid results correctly? Can each technologist correctly differentiate lymphoblasts from reactive lymphocytes? Does each technologist respond properly to no clot detected flags from the coagulation analyzers? Do specimen couriers consistently make their rounds on time? Does Mama wait too long for laboratory reports? Collect data Plan intervention Collect follow-up data Implement intervention We answered these questions by self-measurement. If we found we were performing incompetently at any level, we entered the improvement cycle. The performance improvement cycle (Fig 1) is well suited to upgrading competency. After baseline data were collected, we planned the intervention(s) and implemented it with various improvement tools, such as team building, brainstorming, data analysis, and other techniques described in more detail by the Joint Commission on Accreditation of Healthcare Organizations. 2 Third Tool: Prioritization We had to decide what we should measure to assess competence. In selecting questions related to performance, our team members relied on their own experience as well as complaints, incident investigations, and input from professionals at all levels of our organization. They used this information to design measurements to detect incompetence. Next they selected specific tasks to use in measuring competence, then prioritized the tasks on the basis of complexity, clinical significance, and amount of do-risk. Do-risk is the risk that employees will fail to perform a task, even if they have the necessary knowledge and ability to do it. Do-risk includes the amount of error associated with performing the task itself and the likelihood that employees will disobey policies, take shortcuts, or fail to concentrate or think critically when doing the task (Table). Dorisk can be further understood within the context of the competency hierarchy. Fourth Tool: Know-Can-Do Hierarchy The next step was to decide how to take the measurements. In a report describing physician competence, Norman 3 identified a simple hierarchy of competence, which he called knowcan-do. To be competent, a person must know how to do a task, have the ability to do it, and do it properly without supervision. This model parallels the domains of learning cognitive, psychomotor, and affective. 4 The know-can-do hierarchy is key to designing measurements of competency. The hierarchy is illustrated by the following examples. Know Measurements Measurements at the know level ensure that employees have the knowledge to perform tasks. The College of American Pathologists (CAP) requires verification that technologists know the policies and procedures of their laboratory. We previously used sign-off sheets to document that the technologists had read and understood the manual. Our new strategy, however, called for a more definitive approach. We constructed multiple-choice examinations that focused on highly complex, highly significant, and high do-risk tasks (see Table). In one examination, we asked technologists to use a mock-up hemacytometer to calculate cell counts in cerebrospinal fluid spiked with cells. We searched for answers to 2 questions: (1) Do we follow the counting rules? And, (2) Can we calculate the correct result? The examination helped us to uncover a number of problems. For instance, it allowed us to identify 2 technologists who routinely counted cells in an inadequate number of squares. When we questioned the technologists, we discovered that they did Fig 1. Performance improvement cycle used for upgrading competency. Using this strategy, competency assessment becomes competency improvement. Section 4 Scientific Communications FEBRUARY 2000 VOLUME 31, NUMBER 2 LABORATORY MEDICINE 1 0 7

3 Prioritization Criteria with Tasks High Complexity Flow cytometry testing Correctly gating cell populations for separate analysis Manual cell counting Correctly using counting chamber, determining appropriate dilutions, and calculating results RBC morphology examination Finding and interpreting abnormalities according to established criteria WBC differential counting Finding and correctly interpreting abnormalities associated with blast cells, reactive cells, and other cell types High Significance Clerical work Providing accurate information, assuring patient confidentiality, and treating patients undergoing sensitive testing with professionalism Coagulation testing Following up no clot detected flags* Hematology testing Following up grossly elevated WBC counts* Laboratory result reporting Notifying physicians of abnormal results according to established criteria Phlebotomy Checking patient armbands,* labeling tubes properly, and noting collection sites High Do-Risk Hematology testing Checking edge of blood smear for platelet clumps or other abnormalities Manual cell counting Confirming RBC count with acid lysis Specimen transport Meeting pickup schedules, maintaining documentation Urinalysis Doing confirmation tests, testing for reducing substances on babies *Also meets high do-risk criterion. not understand the counting procedure. The examination also revealed 10 technologists who did not know to confirm RBC identifications with acid lysis. Finally, the examination disclosed 1 technologist who routinely calculated results with the wrong factor. In response to these findings, we revised the cell counting procedure and developed a worksheet with dilution factors and calculation aids (Fig 2). By actively monitoring the worksheets, we could show that the technologists understood and followed the procedures and policies. This procedural examination revealed an opportunity to improve our written procedures, provide continuing education, and accurately determine the level of knowledge of each technologist. Can Measurements Can challenges measure competence in a simulated setting to determine if an employee can perform a task. Direct observations are widely used for this purpose. In our experience, however, direct observations yield little more than knowledge measurements, because subjects usually know that they are being watched, and they properly follow the protocol. These measurements also have questionable value in high do-risk situations. Before 1996, we used color transparencies to create competency challenges in which our technologists routinely performed at near-100% competency. Our experience in 1996, however, showed that this method for competency assessment was inadequate. As a new strategy, we decided to routinely use can measurements to challenge microscopic skills such as reading difficult Gram-stained smears and identifying hematologic morphologies on peripheral blood smears. This required us to prepare patient slides that mimic the highly complex and highly significant aspects of morphology identification. The series of slides consisted of smears associated with various conditions no abnormalities, a bacterial infection with immature granulocytes, a viral infection with reactive lymphocytes, an aleukemic acute lymphocytic leukemia with blast cells and a normal WBC count, and a postoperative patient with increased band neutrophils. In challenging 75 technologists, we found that 17 failed to recognize the lymphoblasts as abnormal cells. With intensive retraining, however, all but 2 established competency. After 3 years of similar challenges and varying our focus to question other skills such as the ability to detect malaria, histoplasmosis, and platelet clumps on the edge of the smear, we have recorded only 1 instance in which a technologist under-reported blast cells

4 The key to improved performance lies in the design of the tool. The transparency method lacked the complexity of the real work situation. Examining a cell with an arrow pointing to it is far different from picking out abnormal cells while scanning a slide in real time. The transparency method assessed only the interpretation of the cell, not the find. Such insensitivity has been shown repeatedly in cytopathology. 5 7 Do Measurements As for do measurements, we questioned whether our technologists were reporting significant RBC morphologic characteristics in peripheral smears. To answer this, we formed an expert oversight panel to retrospectively review reported data and determine our false-negative and false-positive rate. So far we have established new grading criteria based on the number of abnormal cells, measured individual performance with the slide challenge method, and developed a self-assessment study set. The blood cell morphology project involves interventions aimed at all 3 hierarchical levels to improve competence (Fig 3). Although this project is not complete, our goal remains to enhance the value of our morphologic interpretations for quicker diagnosis and treatment. We also hope to reduce the number of shotgun-type anemia work-ups at our medical center. On one hand we wish to deliver meaningful test results, and on the other we want to study how our data is used in patient management. To us, the cost of an unnecessary shotgun-type work-up is a thing-that-mattersto-patients. Summary In our experience, strategies for competency assessment often reveal competence but fail to disclose incompetence. Challenges of low-complexity tasks (such as reading technical procedures) are inferior to challenges that measure understanding and execution of a procedure. Measurements at the do level capture all behavioral domains cognitive, psychomotor, and affective and yield improvements that matter to the patient. The tools we describe are based on common sense. Interestingly, a recent report by the Office of Inspector General, Department of Health and Human Services, 8 on hospital inspection mirrors our view concerning the inadequacy of know measurements and can measurements. The report states that hospital managers not only tell employees what to say during inspections, they even hold practice inspections. Within this culture, inspection becomes a can measurement rather than a do measurement. With proper focus and preparation, the hospital can appear to fulfill the accreditation requirements on the day of inspection. This, of course, provides no indication of how well the hospital performs on other days, thus casting doubt on how well planned inspections ensure quality outcomes. As stated in the report, 8 Joint Commission surveys are unlikely to detect substandard patterns of care or individual practitioners with questionable skills. We similarly conclude that poorly designed competency challenges will probably not detect substandard laboratory performance. Fig 2. Cell count worksheet. Use of the worksheet prompts the technologist to follow and document all procedural steps necessary to obtain a correct result. A supervisor reviews completed worksheets. Section 4 Scientific Communications FEBRUARY 2000 VOLUME 31, NUMBER 2 LABORATORY MEDICINE 1 0 9

5 Fig 3. Our teams rely on standard causeand-effect diagrams to keep focused. This is especially important when working on multifactorial processes that may require an array of interventions to achieve substantial improvement. Standardize morphology grading criteria "Know" projects Feedback performance data from competency challenges Provide updated glossary Continue expert reviews measure Continue periodic measurements measure Improve technical competence Test utilization in evaluating anemia "Can" projects Conduct self study/ Review case studies Provide micrometer measurement tools Continue competency challenges Educate physicians Because patient outcomes depend on competency at all levels of service, we must find ways to detect and correct incompetence.l Acknowledgment The authors would like to thank the administrative and medical staff at the Medical Center of Louisiana at New Orleans. Special thanks to Debbie Fink, Gary Lipscomb, MD, Joyce Majonos, MA, Chuck Surbeck, and Lisa Sackett. References 1. Berte L, Nevalainen D. The laboratory s role in assessing patient outcomes. Lab Med. 1998;29: The Joint Commission on Accreditation of Healthcare Organizations. A Pocket Guide to Using Performance Improvement Tools. Oakbrook Terrace, IL: Joint Commission; Norman G. Can an examination predict competence? The role of recertification in maintenance of competence. Ann CRMCC. 1991;24: Gurvis JP, Grey MT. The anatomy of a competency. J Nurs Staff Dev. 1995;11: Bonfiglio TA, Somrak TM. ASCP education and proficiency testing programs in cytopathology. Lab Med. 1994;25: Hicklin MD, Plott E, Wood RJ, et al. Comparison of written-visual and microslide-screening test measures in cytotechnology. Acta Cytol. 1981;25: Henderson E, Seamans JR, Solomon D. Electronic capture and conversion of microscopy images from cytologic glass slides. Lab Med. 1994;25: Greenleaf JM, Dugan LG, Vey M. The External Review of Hospital Quality, The Role of Accreditation. The Office of Inspector General, Department of Health and Human Services;1999:3. Available at: reports/oei html [accessed July 23, 1999]