Evaluation of the Quality System Appraisal of Conformity

Internal Audit version 061017 Slide 1 Evaluation of the Quality System Appraisal of Conformity Internal Audit Prof. Dr. F. Vanstapel, MD PhD Laboratory Medicine UZ-KULeuven

Internal Audit version 060919 Slide 2 Teaching Goals

Internal Audit version 071130 Slide 3 Teaching Goals - understand auditing as an investment in conformance - understand added value auditing - learn audit techniques - learn to code and report effectively

Internal Audit version 060919 Slide 4 Operational Definition

Internal Audit version 071130 Slide 5 Operational Definition Internal Audit Systematic / independent / documented appraisal of conformity / validity of procedures Systematic evaluation of conformity with the norm-requirements to which we subscribe? Is the audit trail credible? Are the procedures valid = effective? Internal Audit Within and by the organization itself : value-added auditing Are the procedures effective / efficient : Variance with prescribed procedures? Sound / lean / failure-proof design of procedures? Training adequate? System of prevention adequate? Managerial supervision adequate? Proper starting point & end point of this exercise

Internal Audit version 060919 Slide 6 Requirements of the Standards

Internal Audit version 061017 Slide 7 Applicable standards Belgian regulator: KB (3 dec 1999) based on EN 45001 Praktijkrichtlijn updates based on ISO 17025 ISO 15189 Understanding the standard All standards: hierarchical systems defining responsibilities allowing implementation adjusted to the organization allowing a hierarchical document system EN 45001 : main focus on audit-trail & traceability ISO 17025 : + focus on Shewart-Deming -cycle ISO 15189 : + focus on Clinical Practice & Health Economy

Internal Audit version 061017 Slide 8 Requirements of Standards ISO 15189:2003 4.14.1 In order to verify that operations continue to comply with the requirements of the quality management system, internal audits of all elements of the system, both managerial and technical, shall be conducted at intervals defined by the system itself. The internal audit shall progressively address these elements and emphasize areas critically important to patient care. 4.14.2 Audits shall be formally planned, organized and carried out by the quality manager or designated qualified personnel. Personnel shall not audit their own activities. The procedures for internal audits shall be defined and documented and include the types of audit, frequencies, methodologies and required documentation. When deficiencies or opportunities for improvement are noted, the laboratory shall undertake appropriate corrective or preventive actions, which shall be documented and carried out within an agreed upon time. The main elements of the quality system should normally be subject to internal audit once every twelve months. 4.14.3 The results of internal audits shall be submitted to laboratory management for review. Understanding the norm Internal Audit is an exercise in Quality Assurance

Internal Audit version 061017 Slide 9 Understanding the standard The standards are open ended: you determine yourself how to implement it (creative interpretation) The economy of your quality system: - quality = fit for purpose = value for one of the stakeholders - quality = value for your investment = create added value - minimize costs of a lack of quality - maximize return of your investment - weed out all waste steps - lean / failure-proof processes FOCUS ON OPTIMALIZATION OF PROCESSES

Internal Audit version 071130 Slide 10 Do it Right the Maintenance First Time QUALITY INVESTMENT Quality at no Cost A.V. Feigenbaum Repair Investment in CONFORMANCE Costs of QUALITY FAILURE PREVENTION APPRAISAL INTERNAL EXTERNAL Training Maintenance Lean DESIGN for QUALITY End-of-Line Inspection (internal & external) AUDITS Scrap Rework Adverse Effects in Patients Complaints Customer support Action plans ROOT-CAUSE APPROACH Audits address A/B : conformity with norm / procedures * : process excellence

Internal Audit version 060919 Slide 11 Internal vs External Audits

Internal Audit version 060919 Slide 12 Internal Audit can our creative interpretation of the standard * stand scrutiny? vs External Audit credible adherence to standard? Correct? Desired quality? Desired effectiveness? Compliance with standard? Hierarchical system? Audit trail & Traceability? Procedures? Training? Correct? Acceptable quality? State of the art? Corrective and Preventive Interventions? * Auditing refers to compliance with applicable standards Standards and integral quality thinking are covered in a separate module

Internal Audit version 061017 Slide 13 Internal versus external audit External auditor has to stick to the standard: will almost exclusively report on non-conformities examines a limited test scope technical auditor can judge state of the art, & comes with expectation patterns does not know the quality system, thoroughly Internal auditor has to detect non-conformities, but can also report on other issues covers the complete scope is unrelated to the audited section of the lab, and thus lacks expertise & prejudice knows the inner workings of the organization

Internal Audit version 060919 Slide 14 Nuts and Bolts

Internal Audit version 091020 Slide 15 How to generate added value? Audit is an unannounced sample of the reality, suited for analysis of the workings of the integral quality system. (Analyze system with respect to an established standard,* pay attention to system of prevention, management has to act) In an audit the organization can show that it believes in the ability of its personnel to detect shortcomings and in its willingness and ability to remedy these. (Audit can be a motivator) A correctly performed audit is an ideal didactic means to educate the personnel in the workings of the system. (Audit can be a motivator) * Standards and integral quality system thinking are treated in a separate outline

Internal Audit version 070619 Slide 16 BS EN ISO 19011: 2002 Guideline for quality management systems auditing Save time and effort by: Avoid confusion over the objectives of the audit programme Secure agreement of the goals for individual audits Reduce duplication of effort Ensure audit reports contain all the relevant information Ensure audit reports are suited for actions by management Evaluate competence of members of an audit team against objective criteria Quality of scoring - thoroughness / yield - referencing norm-elements - evaluating the importance of non-conformities

Internal Audit version 091020 Slide 17 Murphy s Law Focus of value added Action Plan If it can break, it will break one day. Risk Failures System of Prevention POKA Fail-safe Design for Quality Primary Prevention Universal Lean Protocols Maintenance Schemes Managing Critical Supplies Start-up Inspections Better be prepared! Adversity Focus of value added Audit Secundary Prevention In-line Inspections Intervention Cascades Damage Mitigation Corrective Actions

Internal Audit version 091020 Slide 18 BS EN ISO 19011: 2002 Guideline for quality management systems auditing Save time and effort by: Avoid confusion over the objectives of the audit programme Ensure audit reports are suited for actions by management Role of the hierarchical system Quality System Management integrate information & communicate business goals stay on track facilitate communication dissipate knowledge For maximal effectiveness the quality system should never step in for the management author & actor Workfloor

Internal Audit version 061017 Slide 19 How to audit? Returning ritual : INTRO - The ritual helps to restrain from becoming personal - We are equals, examining together, how to improve our organization - I come as an outsider, with less expertise, please teach me, please take advantage of my lack of prejudice - There are no wrong answers, but there are procedures which have to be improved, & explanations, training that has to be given - Please use the exercise to think now and in the future about how to improve our & your procedures (treat the interviewee as an author)

Internal Audit version 061017 Slide 20 How to audit? Returning ritual : CHECKLISTS - norm based lists - scenario-based lists (available on request) Returning ritual : QUESTIONING - total vs sampling - horizontal vs vertical - closed vs open R) a mixture of all techniques

Internal Audit version 061017 Slide 21 Interview techniques Open questions: allow the interviewee to become an author of ideas Stop supervisors intervening in the discourse of the technicians Keep interview going by grouping items Do not go for a total audit, but move vertically, when you find a non-conformity Note taking should not interfere with the interview Produce a report that invites for a constructive response

Internal Audit version 061017 Slide 22 While you listen, continuously ask yourself Traceability Audit trail: Can results be traced to personnel, reagents, calibrators,? Training: Does technician operate on common sense or on knowledge? Quality of supervision: Do supervisors leave traces of their interventions? Communication: Are complaints, evaluation of suppliers, etc. processed by the organization? Quality of interventions: Are corrective actions expanded to collateral damage? Are preventive actions undertaken? Are they evaluated? Validity of procedures: Norm-compliance? Argued/documented in the test validation file?

Internal Audit version 061017 Slide 23 How to audit? Returning ritual : PLANNING - on short notice - no shuffling - better many short then a few long audits Returning ritual : REPORTING - be fast while you can still reconstruct your short-hand notes while interviewee still remembers the interview - be factual and precise (procedures, reagents, tests) - refer to elements of the standard & grade non-conformities - recognize the interviewee as author of ideas, and identify yourself as author of opinions: this is important to sustain mutual trust

Internal Audit version 070620 Slide 24 CODING FLOWCHART IDENTIFY ITEM RECORD FINDINGS + +* NO NO Sporadic Patiënt Safety Repeating B NO B Remark? Frequency? A YES Standard/Procedures Patient-safety? YES YES Systematic ALL ITEMS REQUIRED Analyse ± actie Adjust Processes Greatest Added Value Immediately Deploy Procedure

Internal Audit version 061017 Slide 25 How to audit? Returning ritual : ACTION PLANS registered by a single person, but authored by many - be fast while you can still remember the interview - be factual and precise (refer to the non-conformities of the report) - start with analysis by the person at the origin of the problem - let that person become the author of the solution - always indicate action, deadline, and responsibilities Returning ritual : FOLLOW UP - transfer follow-up of action to a higher organ and simultaneously feed the management review process - use repeat audits as a means of follow up

Internal Audit version 060919 Slide 26 Summary

Internal Audit version 061017 Slide 27 Take home message The content and topics of the audit relate to the daily supervision & the efficacy of management The ritual allows to perform an audit without becoming threatening Systematic questioning allows to recognize blind spots of personnel, supervisors & management Overall the audit creates value by assisting in improved management by stressing that quality is realized on workfloor / responsibility of coworkers

Internal Audit version 061017 Slide 28 Aids Know the standard * Know your implementation of the standard: = your quality system the document system, the internal quality assurance procedures, the validation procedures, Checklists Introduction to the audit Evaluation form (hard-copy versions can be obtained upon request) * The standard is treated in a separate module

Internal Audit version 061017 Slide 29 Recommended literature Standard ISO 15189:2003 (CEN Comité Européen de Normalisation) BELAC 6-102 Checklist for evaluation of conformity towards requirements of the standard NBN EN ISO 15189 BELAC 2-102 Toepassing van de accreditatiecriteria: bijkomende interpretatie voor de sector klinische biologie CPA 0204 An approach to audit in the medical laboratory