Expert Panel on Diagnostic Imaging Quality

Transcription

1 Expert Panel on Diagnostic Imaging Quality Report Draft for Expert Panel Review and Input Design Recommendations for Ontario Peer Review for Diagnostic Imaging

2 Table of Contents Executive Summary Introduction Peer Review in Context Peer Review Program Design Framework Context for Design: Jurisdictional Scan and Literature Review Peer Review Design Recommendations Peer Review Implementation Approach Recommended Next Steps Conclusion Appendix A: Expert Panel Terms of Reference Appendix B: Panel Meeting Dates Appendix C: List of Jurisdictions and Articles Reviewed List of Articles Reviewed Appendix D: Data Tables Appendix E: Sample Scoring Approaches Appendix F: Sample Reports Glossary Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

3 Executive Summary Since 2010 with the introduction of the Excellent Care for All Act and Strategy, the Ontario government has taken a number of steps to improve the quality of Ontario s health care system and enable the system to provide the best possible patient care. There have been a series of new standards developed and implemented to ensure that Ontarians receive health care of the highest possible quality and value. Work over the past decade to improve patient safety has largely focused on addressing issues such as medication errors, health care associated infections, and postsurgical complications. Diagnostic error has received comparatively less attention. Increasingly, a number of studies have demonstrated the importance of focusing quality efforts in the area of diagnostics.1 This work builds on existing activity in provincial quality management with the long-standing program in diagnostic pathology and current development of a provincial quality management program for colonoscopy, mammography and pathology. Overall, over 20 million diagnostic imaging procedures The surrogate indicator of radiological excellence that has become accepted is consistency of assessments between radiologists, and the technique that has become the standard for evaluating concordance is peer review. ~ A workstation-integrated peer review quality assurance program: pilot study. O Keeffe et al. BMC Medical Imaging 2013, 13:19 are delivered each year in Ontario hospitals and Independent Health Facilities (IHFs) by fully qualified diagnostic imaging specialists responding to requests from other physicians and other authorized health professionals. These important services are a vital element in the diagnosis and follow-up of a broad range of diseases and patient care and are delivered at a high level of quality. A recently reported event in Ontario and events in other jurisdictions of misdiagnoses in radiology reaffirm the critical role of quality assurance in the delivery of excellent patient care. While a certain degree of error is inevitable as the provision of diagnostic services often involves making decisions under conditions of uncertainty, it is necessary to identify the regulatory and statutory safeguards already in place and where applicable, any further formal measures that are needed to reassure the public and support physicians and the organizations they work in to create the conditions necessary to assure and improve quality of care. These situations are a useful reminder of the need to re-examine current approaches and to identify new ways to ensure improved accuracy that benefits patient care and safety. Many aspects of diagnostic radiology make it amenable to objective performance measurement and improvement. The field of radiology is largely digital, with groups of imaging physicians who are accustomed to teaching and learning from one another. The discipline has also demonstrated its ability to adapt to rapidly evolving technology. However, in designing the approach to quality management in diagnostic imaging, it is important to ensure that the program is designed in a way that leads to meaningful performance improvement rather than simply tracking individual error rates. It becomes a question of whether to study the what, when, and how of an event or to simply focus on the who. 2 1 Diagnostic Errors The Next Frontier for Patient Safety. David E. Newman-Toker, MD, PhD Peter J. Pronovost, MD, PhD 2 Rethinking Peer Review: What Aviation Can Teach Radiology about Performance Improvement. David B. Larson, MD, MBA, and, John J. Nance, JD. Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

4 Against this backdrop, on December 3, 2013, the Ontario Minster of Health and Long Term care requested that Health Quality Ontario (HQO) and its health partners lead the implementation of a province-wide physician peer review program in all facilities where diagnostic imaging services are provided, including mammograms and CT scans. An Expert Panel was convened to develop recommendations for the design of this Peer Review Program. Consultants from PricewaterhouseCoopers provided support to this Panel by developing meeting materials, documenting and synthesizing Panel discussions and conducting a review of the literature and practices in other jurisdictions as inputs to design. The Canadian Association of Radiologists (CAR) describes Peer review as a generic term for a process of self-regulation by a profession or a process of evaluation involving qualified individuals within the relevant field. Peer review methods are employed to maintain standards, improve performance and provide credibility. A peer review process in diagnostic imaging is typically used in the context of a radiology service s overall quality assurance program. Peer review is ideal for measuring radiologists skills in the regular daily workflow, particularly in a digital environment as it essentially evaluates the end product of a radiologist s work by having a colleague reviewer correlate an exam with his/her peer s report. There is significant educational value when the identification of discrepant cases are identified to become active learning and quality assurance opportunities for individual and group improvement in order to minimise repetition. The following outlines the Expert Panel s recommended Program goals and recommended approach to Peer Review in Ontario. Ontario Diagnostic Imaging Peer Review Program Goals Enhance the consistency and accuracy of diagnostic imaging services to improve quality of care for patients Support ongoing improvements to diagnostic image interpretation skills through peer-topeer learning in a non-punitive environment Enable informed decisions about patient treatment, enhancement of quality programming, physician training and continuing medical education Support maintenance of ongoing learning, education and contribution to a culture of quality improvement, transparency and accountability 4 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

5 Recommended Approach to Peer Review in Ontario Peer review is education and learning focused and is part of an overall quality management program. Peer review is part of a broader framework of quality management processes and tools and is not meant to be the definitive stand-alone quality assurance mechanism for a facility or the system. The Ontario peer review program will adhere to the guidelines set out by the Canadian Association of Radiologists (CAR) on Peer Review. Peer review elements will be consistent across the Province but the implementation approach will take the local infrastructure and needs of the facilities into account The Program will be implemented at the facility level but facilities can work in collaboration. A certain critical mass of radiologists/interpreting physicians radiologists) and infrastructure is required to support a peer review program, which will require facilities to collaborate in some cases. The program will leverage existing local and provincial technology systems for peer review. No single IT system should be prescribed to support peer review. Existing provincial diagnostic image repositories should be leveraged to support implementation of peer review. A balance must be struck between the protection of health care professionals to participate openly in quality assurance activities and the requirement to protect patient safety in a case where the quality assurance activity surfaces a potential patient risk. The appropriate legislative and regulatory framework must be in place, supported by consistent application and implementation across all facilities where diagnostic image interpretation is taking place. o This recommendation may be further developed as part of the review of the Quality of Care Information Act (QCIPA) currently underway with expected recommendations by the end of QCIPA protection should be applied consistently to all participating in this program regardless of facility type. A regulatory change may be required to include IHFs. The needs of smaller and more remote facilities and varying facility types will need to be considered as part of implementation to ensure that the program is fully implemented. The following are the highlights of the recommended program design. These design recommendations reflect the recommended approach and the best experience and evidence to date. However, peer review is an important quality assurance tool for diagnostic imaging services, and should be seen as being a key part of a quality management framework and its implementation and evaluation should be considered within this broader context. Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

6 Overview of Peer Review Program Design Recommendations Sampling and assignment will be random, representative of radiologist work and will be peer matched accordingly. Peer review may be prospective (before a case is finalized) or retrospective but time limited (within a relatively short period of time from report finalization to enable intervention in patient care if necessary). Confidentiality is required for all aspects of peer review. Anonymity between reporting and reviewing physician may be of added value as part of the peer review process. For the purposes of learning and education, anonymity of the reporting and reviewing physician is essential however, that anonymity will be limited to the peer review program. A consistent approach to scoring and providing feedback as part of the peer review is recommended. Education and training on how to do peer review will be required to ensure consistent application. A four point system is recommended aligned with the American College of Radiology s (ACR) RadPeer scoring approach with consideration of the inclusion of a separate classification or score for good catches. A local quality improvement review committee is required to review cases and address possible significant discrepancies that impact clinical management and outcome. The Chief/ Quality Advisor may or may not be part of this committee but is accountable for ensuring the effective operations of the committee, the decisions of the committee and management of issues that may arise. Peer review programs will be supported by the development of local policies, procedures and related communication to patients and the general public to meet their information needs and support the goal of transparency overall. Taken together, the program goals and design recommendations provide a strategic direction and framework for the implementation of peer review. While more work is needed to support the next phase toward implementation, the group identified the following key implementation planning principles to guide the next phase of work. These principles highlight the importance of a focused and practical approach that takes the broader context of health system transformation into account and the need to build on, and learn from, implementation experiences. 6 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

7 Recommended Peer Review Implementation Principles Phased and iterative Aligned with related initiatives and as part of an overall quality management program Consider impact, risk and readiness in developing implementation phases Supported by appropriate and sufficient resources, tools and infrastructure These recommendations will support the refinement and detailed planning of existing and proposed peer review programs for imaging in the province. The recommendations also provide the foundation for a more focused implementation effort across all facilities, providers and modalities where peer review does not exist today. As part of the next phase of work, it is recommended that relevant select sections of this report be broadly shared to create a common understanding of the goals and principles and design recommendations of peer review across the Province and to lay a foundation for implementation. Implementation planning is a complex and multi-faceted endeavor. It is recommended that this Expert Panel continue its work with the support of a Working Group to conduct the necessary analysis and deliberation to outline detailed design elements, a specific implementation plan and timeline as well as associated costs and enablers. Finally, the Expert Panel is excited to begin exploration of additional components of a QA program for diagnostic imaging starting with accreditation and would propose to do so as part of the next steps. Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

8 1. Introduction In an announcement on December 3, 2013, the Ontario Minister of Health and Long Term Care requested that Health Quality Ontario and its health partners lead the implementation of a province-wide physician peer review program in all facilities where diagnostic imaging services are provided, including mammograms and CT scans. Following this announcement, an Expert Panel was convened (see Appendix A for terms of reference and membership) to develop a recommended design and approach to implementation for peer review that is focused on patient safety and supporting the delivery of consistent, high quality care. The short term goal of the expert panel was to focus on peer review. This work was initiated with an understanding that peer review is a key component of a broader QA program for diagnostic imaging (DI) and that the outcomes of this phase and future phases could potentially serve as the conceptual framework of a broad based provincial QA program that could apply to other areas of clinical practice beyond DI. This work builds on and aligns with other provincial quality management efforts, including programs in diagnostic pathology and the Quality Management Program (QMP) being co-led by Cancer Care Ontario and the College of Physicians and Surgeons of Ontario for mammography, colonoscopy and pathology. The Expert Panel met over a series of nine meetings from December 2013 through July 2014 (see Appendix B for meeting dates and Panel presentations) to work through the recommendations for a peer review program in Ontario. Supported by consultants from PricewaterhouseCoopers, members worked through a structured and iterative process to review the global literature, understand the Ontario context and then define program goals, principles, core design elements as well as recommend next steps. The literature review and jurisdictional scan documents are available as separate attachments to this document. This report summarizes the findings and recommendations of the Expert Panel for a Diagnostic Imaging Peer Review Program in Ontario that supports consistently high quality patient care. These recommendations reflect the consensus position of the Expert Panel and largely build on the recommended guidelines for peer review systems published by the Canadian Association of Radiologists (the CAR Guide to Peer Review Systems published 2011, amended 2012) as well as the published literature on peer review programs and the experience developing and implementing peer review in Ontario and other Canadian and international jurisdictions. The report also provides high-level recommendations on the approach to detailed design and implementation planning which, is an essential next phase in program development. The objective of knowledge translation is at the core of a peer review program in diagnostic imaging. Such learning is achieved from the full spectrum of diagnostic imaging good catches through to interpretive errors. While education is the only objective of peer review, it is recognized that at the local/facility level there may be cases that fall outside the norm of diagnostic accuracy, which are identified through the peer review program. The profession recognizes that processes need to be in place and steps need to be taken to address these outlier cases. It is important to state that dealing with outside the norm instances is not the role of peer review, but that as part of this report the need to document how such cases are dealt with at the local/facility level is included as a component of a comprehensive quality program. The recommendations outlined in this report could act as discussion points for consultation with the broader diagnostic imaging clinical quality stakeholders in Ontario. Both a broad-based 8 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

9 consultation process for the recommendations of this document and a well-constructed process for detailed design and implementation planning are critical success factors for the implementation of peer review in Ontario. 2. Peer Review in Context Peer Review Defined The definition and purpose of peer review is challenging to communicate, having different meanings to different people. One of the most important recommendations of the Panel s work is that there be a consistent and shared understanding of what peer review is, its goals and what it is meant to accomplish as part of an overall quality management framework. An effective program for peer review should reveal opportunities for quality improvement to help ensure competence and help improve patient outcomes. Cases ideally, should be selected at random to broadly represent the work performed in the radiology department or facility. The evaluation process should be consistent, with all personnel being aware of, and adhering to established rules and procedures. The process also should be timely in order to represent the current state of performance and interpretations should be evaluated within a reasonable interval after the initial report. Peer review should be ongoing so that data can be tracked over time and analyzed to reveal trends. One of the most important success factors of any quality assurance program is participation. As with any part of a performance evaluation process, to encourage full and effective participation, peer review should be non-punitive, have a minimal effect on work flow, and allow easy participation. 3 This Panel endorses the definition of peer review according to the Canadian Association of Radiologists peer review is a generic term for a process of selfregulation by a profession or a process of evaluation involving qualified individuals within the relevant field. Peer review methods are employed to maintain standards, improve performance and provide credibility. 4 Peer review can either serve as a coach or as a judge, but it cannot successfully do both at the same time (and it has not been shown to do the latter very well in any case) ~ Rethinking Peer Review: What Aviation Can Teach Radiology about Performance Improvement (Larsen et. al) The type of peer review program envisioned for Ontario is one that takes place at the workstation, occurs between physicians, and supports the learning and education of all physicians reading images regardless of the diagnostic imaging modality or location. The focus on learning and education cannot be underestimated and is essential to the success of this program. The features of peer review are what distinguish it from other quality assurance processes. According to the CAR s 2012 Guide to Peer Review Systems (p.13), features of peer review include: The process includes a reactive or proactive double reading with two physicians interpreting the same study 3 Peer review in diagnostic radiology: current state and a vision for the future. Mahgerefteh S1, Kruskal JB, Yam CS, Blachar A, Sosna J. Radiographics Sep-Oct;29(5): doi: /rg Epub 2009 Jun The CAR Guide to Peer Review Systems published 2011, amended Review.pdf Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

10 The process allows for the random selection of studies to be reviewed on a regularly scheduled basis Examinations and procedures are representative of the work of each physician s specialty The process allows assessment of the agreement of the original report with subsequent review (or with surgical or pathologic findings) There is an approved classification of peer-review findings with regard to level of quality concerns (e.g. a 4-point scoring scale) Policies and procedures for action to be taken on significantly discrepant peer-review findings are in place for the purpose of achieving quality outcomes improvement Summary statistics can be generated and comparisons shown for each physician by modality to help the coordinator assess performance standards Summary data for each facility or practice by modality can be obtained to aid the departmental QA program There should be a planned strategy for remediation and re-education on both individual and departmental levels when discrepancies arise Another important element of peer review is the establishment of a local committee to provide oversight to the process. This committee is to be comprised of physicians with an appointed chair who may or may not be the Chief or Quality Advisor. This committee supports the development and implementation of the peer review program locally and supports the review of significant discrepancies as well as acting as a guide for the overall learning and education processes that are key to peer review success. Ultimate accountability for the program locally however rests with the qualified imaging physicians acting in their capacity as Department Chief (hospital) or Quality Advisor (IHF) regardless of whether he or she is a member of the Committee. The current variability of the understanding of what peer review means was demonstrated by the survey that was developed and distributed as part of the work of this Panel. In May 2014, hospitals and IHFs were sent surveys in support of this initiative. The purpose of the survey was to assess the state of adoption of QA and specifically peer review programs for diagnostic imaging in these facilities as well as to gain information on volumes and services. According to the survey results, while both some hospitals and IHFs report the adoption of peer review programs, the features of most of these programs do not appear to align with the elements of peer review as recommended by the Canadian Association of Radiologists Guidelines for Peer Review. However, these practices reflect the significant receptivity in the field to pursuing peer review and represent a strong foundation upon which to build while moving toward a consistent approach to future peer review implementation. Ontario radiologists have led efforts to introduce peer review into diagnostic imaging practices. Peer review systems have only recently been facilitated by the advent of digital imaging systems like PACS (Picture Archiving Communication Systems) and RIS (Radiology Information Systems) and the increased deployment of digital imaging technologies. The development and availability of electronic peer review systems is a relatively new occurrence in diagnostic imaging environments. 10 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

11 Volume and Location of Service Delivery The fact that imaging services today are delivered in both academic and community hospitals as well as IHFs represents an important characteristic of the Ontario context. The recommendations of this report support the delivery of services where they are delivered today with a process for peer review that is consistent and supports the overall culture of quality improvement. As part of implementation planning, it will be necessary to address scale and infrastructure in smaller hospitals and IHFs to facilitate full participation and coverage. The following data highlights service volume, modality and facility data analyzed for radiology services in Ontario based on administrative data for FY 12/13 received from the Ministry of Health and Long Term Care Health Analytics Branch. Approximately 60%, of diagnostic imaging is done in hospitals where the majority of CT and MRI scans are delivered. It is important to note that this data does not definitively represent the volumes and locations particularly as it relates to IHFs where the current data does not include IHF CT and MRI volumes (estimated to be relatively low) and existing data structures make it more challenging to calculate site-based volumes. 5 As part of implementation planning, additional efforts will need to be made with Ministry partners to conduct a more detailed analysis. However, this analysis presents an order of magnitude assessment of the total volumes and proportion delivered in hospitals as compared to IHFs. Figure 4: Radiology Volumes by Location MOHLTC Data (Fiscal Year ) Total Volume of Radiology Services 21,982,307 Annual Hospital Volume 12,507,972 Annual IHF Volume 8,190,749 2% 4% FY % 57% Annual Hospital Volume Annual IHF Volume Annual Office & Private Lab Volume N/A 5 Note: Due to the caveats with the data provided, care must be taken in drawing conclusions related to the landscape of DI services The data received from the Ministry is based on the physician billing codes and thus include both in-patient and out-patient service volumes with hospitals The data received from the Ministry does not have any service volumes for CT & MRI services provided by IHFs The data includes entries with location type as Not Assigned, which is due to the incomplete information filled in the forms when the services were provided The IHF data provided is based on per site service volumes. However, it has been observed that the volume is not the true representation of the sites. An attempt has been made to collate the data for the sites with the same corporate clinic names The Ministry data does not validate the ownership structure of the IHF sites which may lead to the lack of representation of certain organizations operating multiple IHF sites with different license names. Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

12 A significant percentage of volumes of services are represented by the top ten facilities. There is significant overlap among the top 10 hospitals performing the majority of CT and MRI and Ultrasound and x-ray. In both cases, the top three are University Health Network, Credit Valley and Trillium (now Trillium Health Partners) and Sunnybrook Health Sciences Centre. In hospitals, the top 10 volume facilities represent close to 40% of all hospital-based diagnostic imaging. X-ray and Ultrasound are the key services offered by the IHFs. The top 10 volume IHF sites represent just over 40% of all volumes of x-ray and ultrasound performed in IHFs. Details of the volumes by modality, facility and the top ten sites for each can be found in Appendix D. Peer Review as Part of a Broader Quality Management Program It is also important to note that peer review is a key element of a comprehensive QA framework. Figure 5 illustrates peer review in the context of a broader quality framework. As such, it is important to understand that peer review alone will not assure quality. As one of several QA tools, it will support the development of a culture of continuous quality improvement and contribute to the implementation of broader quality frameworks. Peer review must be developed according to the principles and recommendations outlined in this report and it must be implemented within the context of a broader quality management program. In addition to peer review being part of a broader quality management framework, the interpretation of diagnostic images is also part of a chain of activity related to patient safety and quality of care that includes technical issues, communication, pathology, referring doctors taking action, radiology technicians etc). Figure 5: Peer Review in the Context of a Quality Management Framework Framework adapted from Quality Management Program (QMP) framework. Cancer Care Ontario/ College of Physicians and Surgeons of Ontario with input from G. Ross Baker and associated literature. 12 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

13 3. Peer Review Program Design Framework The Panel worked through a structured approach to design outlined in Figure 6. The design activity was based on a foundation of evidence, other practices and an understanding of the characteristics of diagnostic imaging services in Ontario today and existing peer review programs both in hospitals and IHFs. Figure 6: Ontario DI Peer Review Model Phase 1 Design Framework The design recommendations made by the Panel follow the core processes involved in peer review. These processes and their definitions are outlined below. Figure 7: Peer Review Processes Peer Review Process 1. Sampling and Assignment 2. Review and Provision of Feedback 3. Case Review and Discussion Definition Select and deliver cases for peer review to participating physicians Score case and make notes related to score Collaborative review of selected peer review cases for learning and education amongst physicians 4. Learning and education Generate new knowledge and/or skills to improve quality 5. Measurement and Reporting 6. Management of Significant Discrepancies Quantify and/or qualify activities and share results to improve quality Flag and address cases that reflect possible issues that may have a negative impact on patient care Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

14 4. Context for Design: Jurisdictional Scan and Literature Review Evidence and other practices informed the design of a Diagnostic Imaging Peer Review program in Ontario. As part of this phase of work, a jurisdictional scan and two separate literature reviews were conducted by PricewaterhouseCoopers to provide input to the work of the Panel. The jurisdictional scan included a review of seven peer review programs across North America, each with programs that vary based on operating models and stage of development. The initial literature review covered 15 articles from an initial list of 56. The articles were filtered based on impact of the program relevance to the Ontario context (see Appendix C for list of jurisdictions and articles). In addition to this work, the Ministry of Health and Long Term Care s Heath System Strategy and Policy Division, at the request of Health Quality Ontario, conducted a detailed review of the literature on peer review, accreditation, and quality assurance practices in relation to diagnostic imaging published between 2010 and Over 9,000 scholarly and grey literature publications were identified based on the inclusion criteria. After The call for peer review to be proactive, educational and nonpunitive is supported by reasons that are advanced in the literature. ~ Ontario Ministry of Health and Long Term Care. Literature Review on Diagnostic Imaging: A Report on Discrepancy and Error, Peer Review, Accreditation and Quality Assurance June 18, 2014 refining the literature search and assessment, 89 publications of clear relevance were included. The literature reviews and jurisdictional scans are presented as companion documents to this report. Both the review of other jurisdictions and the literature scans demonstrate that peer review programs are still evolving with few standards or established leading practices in health care systems. Ontario has an opportunity to contribute to the body of evidence as it advances a peer review program. Evaluation and reporting of any provincial initiative is a recommended part of the opportunity. 14 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

15 The guidance provided by these reviews across several elements of a peer review program improvement is summarized below. Figure 8: Synthesis of Guidance from Jurisdictional Scan and Literature Review EDUCATION A peer review program should promote an environment of learning and improvement for imaging physicians by creating a platform to communicate with their peers Peer review programs with a participation based focus are seen to have a higher adoption rate GOVERNANCE Peer review processes must be unbiased and balanced Grading scales (qualitative and/or quantitative) are used to audit and validate data and create a repository for future references by imaging physicians Independent quality committees made up of imaging physicians need to be established for the data review and discrepancy resolution Local imaging physician groups should be responsible for setting up the processes for individual physicians based on the standards provided, with independent committees overlooking those departmental processes TECHNOLOGY CT and MRI have been the key areas of interest for the implementation of the majority of peer review programs Digital systems such as PACS and Radiology Information Systems (RIS) are key enablers for the peer review program implementation Larger scale and participation of multiple organizations across a geography can lead to the development of a centre of excellence in radiology Data availability and transparency are frequently used as motivational tools to improve performance but need to be carefully designed & implemented ACCREDITATION Accreditation can be an important lever for the adoption of the peer review program at the facility level Currently accreditation is a lever being used by the American College of Radiology (ACR) to maximize adoption of the peer review program in radiology in the United States Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

16 5. Peer Review Design Recommendations The design recommendations reflect the key values and priorities of the Program and are essential ingredients for implementation planning. Getting the principles right is essential to the success of the program. The design principles define and communicate the key characteristics of the product or program to a wide variety of stakeholders including clients, colleagues, and team members. The following are the principles developed and adhered to by the Expert Panel in developing design recommendations. Together with the program goals, they become the foundation upon which the program is built and delivered in Ontario. Principles of Ontario s Peer Review Program Peer review should identify opportunities for quality improvement, facilitate improved outcomes, and contribute to increased competence. Review of possible errors made by colleagues is a recognized learning opportunity for the reviewing physician, the interpreting physician, and those participating in discrepancy rounds or related educational activities. ~ A workstation-integrated peer review quality assurance program: pilot study O Keeffe et al. BMC Medical Imaging 2013, 13:19 Principles of Ontario s Peer Review Program Integrated within a broader quality framework Details Peer review is one of a number of quality assurance tools that supports maintaining standards and improving quality Standards-based The peer review program will adhere to Canadian Association of Radiologists (CAR) principles for peer review Provincial consistency and coverage Peer review elements will be consistent across the province but the implementation approach will take the infrastructure and needs of the facilities into account The program will be implemented at the facility level but facilities may need to work in collaboration to meet the standards of the program Applies to all interpreting physicians, all facility locations, all modalities Learning and education focused The program is focused on education and learning, and contributes to an overarching quality educational agenda The provision of feedback and education among peers is intended to improve overall learning within the profession 16 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

17 Principles of Ontario s Peer Review Program Details Accountable The program clearly defines roles, responsibilities and accountabilities The program is supported by a consistent regulatory, medico-legal and privacy framework Sustainable The program must be cost effective and efficient to implement and administer The program leverages existing local, regional and provincial resources The program recognizes that various discrepancies exist between sites that may require resource allocations The goals of the program represent the outcomes against which program success should be evaluated. Program Goals Enhance the consistency and accuracy of radiology services to improve quality of care for patients Support improved diagnostic image interpretation skills through peer-to-peer learning Enable informed decisions about patient treatment, enhancement of quality programming, physician training and continuing medical education Support maintenance of ongoing learning, education and contribution to a culture of quality improvement, transparency and accountability in a non-punitive environment Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

18 The program design recommendations are outlined below according to each of the program subprocesses described in section 3, Peer Review Program Design Framework. Program Design Recommendations Program Design Elements: Sub Processes 1. Sampling and Assignment 2. Review and Provision of Feedback 3. Case Review and Discussion Recommendations Peer review may be prospective (before a report is finalized) or retrospective (after a report has been submitted). If the sampling is retrospective, it should be time limited (close to the final submission date) in order to maximize patient benefit. Sampling and assignment should be random, representative of radiologist work and should be peer matched accordingly. Confidentiality is required for all aspects of peer review. Anonymity is required for cases reviewed for the purposes of learning and education. Anonymity between reporting and reviewing physician may be of added value. A certain degree of scale (number of physicians) and infrastructure is required to support sufficient sampling that may require facilities to collaborate in some cases. A consistent and timely approach to scoring and providing feedback as part of the peer review is required. A four point system is recommended aligned with the ACR RadPeer scoring approach with possible additional inclusion of a separate classification or score for good catches where difficult or subtle findings have been uncovered. These cases are of high teaching value. Examples of scoring systems may be found in Appendix E. Education and training on how to do peer review is required to ensure consistent application. Provision should be made to add notes to describe feedback as a companion to the score. Cases will be reviewed locally at the facility/ department level on a regular basis. A local quality improvement review committee will review cases and may also develop the learning and education approach for the facility or network (per CAR guidelines). The Chief/ Quality Advisor may or may not be part of this committee but is responsible for ensuring the effective operations of the committee, actions of the committee and is ultimately accountable for the decisions of the committee or any issues that arise. A balance must be struck between the protection of health care professionals to participate openly in quality assurance activities and the requirement to protect patient safety in a case where the quality assurance activity surfaces a potential patient risk. There is an existing legislative and regulatory framework governing the delivery of diagnostic imaging services in place and this must continue to evolve to address and incorporate changes important to quality and patient safety, and be supported by consistent application and 18 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

19 Program Design Elements: Sub Processes 4. Learning and education Recommendations implementation across all facilities where diagnostic image interpretation is taking place. This recommendation may be further developed as part of the review of the Quality of Care Information Act (QCIPA) currently underway with expected recommendations by the end of Physicians should be notified of their peer review scores. Significant discrepancies will be flagged for the local review committee immediately for follow-up. The local quality improvement review committee is responsible for determining the appropriate follow up once a finding of a significant discrepancy has been confirmed. Learning and education is the primary focus for the peer review program. Structured educational rounds/ peer review conferences derived from the case review process should take place on a regular basis. Physicians are required to participate in these educational activities. Learning points should be documented from these activities and reported on and shared. 5. Measurement and Reporting Reporting on number of cases and scores may be provided at the physician level and facility level to support peer review program management. Reporting at the provincial level is important to ensure transparency and demonstrate the efforts being made at the physician and facility level to support quality assurance. Provincial reporting will be under the oversight of Health Quality Ontario, an arm s length organization and such reporting may include: Names of facilities and number of physicians in each participating in the Ontario peer review program (features outlined here) Number of Peer Review conferences and number of cases reviewed in each facility annually with % attendance and summary of key learning points Measurement and reporting are important features of accountability. Exact details of the accountability framework and reporting model will be defined as part of detailed design and implementation planning and oversight for implementation and ongoing reporting will be the responsibility of Health Quality Ontario. See Appendix E for possible report samples. 6. Management of Significant Discrepancies Although the purpose of peer review is for ongoing education and learning, there is the potential that on occasion peer review may surface issues that require follow-up with the patient and the interpreting physician. Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

20 Program Design Elements: Sub Processes Recommendations In cases where a significant discrepancy is identified, there may be a requirement to ensure that a prompt addendum to the original report is issued and that disclosure to the patient is made conforming to clinical and professional standards. The validation of a finding of a major discrepancy by the local quality improvement committee requires the Chief/ Quality Advisor to determine the appropriate physician follow-up, considering this finding, a pattern of findings, and possibly other factors known to them. Additional follow-up may include education and learning or referral out of the peer review program for follow-up under an alternate process. Further review of the clinician's performance beyond the peer review finding may be the first next step. In all instances patient care is the primary focus, and clinical practice guidelines and standards must be consistently adhered to. In the event a concerning situation arises the obligation is that the review committee will promptly send the case outside the peer review program for appropriate follow up with the Chief/Quality Advisor. The terms of reference of the local review committee need to clearly outline their responsibility in this regard. The development of the local peer review program policies and procedures represents an opportunity for each facility to review the related statutory and regulatory requirements and to reinforce all quality practices. It is expected that facilities have established processes and mechanisms for dealing with cases that fall outside the norm. As part of the implementation planning phase of this initiative, policies and mechanisms will be identified (and documented) at each facility. Assistance may be required at the local level to set out or implement such policies. To address such a need, this Panel will develop a toolkit of specific guidance for addressing concerning issues beyond peer review. Existing local approaches to involving such bodies as the Medical Advisory Committee, the Board and other advisory bodies will continue to be mechanisms through which significant discrepancies will be addressed, and where required, be reported to the CPSO. 20 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario

21 6. Peer Review Implementation Approach The transition from program design to implementation is a critical juncture at which ultimate success is often determined. The next phase of work will include development of guidance on the details of program design and planning for implementation. Elements of detailed design such as the percent of cases to be sampled and the critical mass of volumes and providers required to support the principles outlined in this report will be identified where the evidence is sufficient to support it. In cases where evidence is not sufficient, a consensus approach will be used to provide guidance in key areas. Most importantly, the next phase of work will include more detailed data analysis and consultation with facilities and providers in order to outline a specific plan and associated costs for the implementation of peer review in Ontario. Quality improvement requires a careful, dedicated, and continuously planned effort by a number of skilled and committed team members, with the goal being to do the right thing in a timely fashion in every case. ~ Kruskal, JB et al. Quality initiatives: Quality improvement in radiology: basic principles and tools required to achieve success. Radiographics Oct;31(6): doi: /rg The next phase of implementation planning will address the complexity of developing and implementing quality assurance programs and outline the specific leadership, resources, infrastructure, tools, technologies and other key enablers required to successfully implement this program. The complexity associated with peer review processes is largely associated with the development of a sustainable culture of continuous quality improvement across a diverse and significant number of facilities. The implementation of peer review will need to contend with the issues associated with moving toward consistent approaches to quality improvement in general and will not be specific to peer review alone. As such, it will be important to engage with a wide variety of partners and stakeholders to ensure alignment and continued support. It is also important that the implementation activity to follow this report take into consideration the criteria for the evaluation of success. The ability to qualify and quantify implementation success criteria will serve as a guide during implementation and also support more formal evaluation activity if Ontario elects to contribute to the growing body of evidence in the literature. The following outlines the recommended approach to planning and managing the implementation of these recommendations. Below is a set of implementation principles to be adhered to in the next phase of work. Implementation Principle Phased and Iterative Details Organize implementation into phases where each phase incorporates learning from previous phases Aligned with related initiatives Where appropriate, program implementation must align with other related initiatives, timing and focus (e.g. CPSO Peer Assessment, QMP Mammography, eho Diagnostic Imaging Repositories, CPSO IHF Clinical Practice Parameters) Peer review should be implemented as part of an overall quality management program Health Quality Ontario Expert Panel on Diagnostic Imaging Quality

22 Implementation Principle Consider impact, risk and readiness in developing implementation phases Details Provide support to all local initiatives ready to implement in adherence with this design standard Focus on high risk modalities (CT and MRI) and larger volume centres where education and learning can have large-scale impact Proceed quickly to connect in smaller and/or more isolated centres to achieve greater coverage geographically Identify key readiness criteria and look to leverage existing initiatives to ensure key factors are in place Special attention and support will be required to address the needs of smaller and/or more remote locations Supported by appropriate and sufficient resources, tools and infrastructure Ensure that implementation includes the infrastructure required to support the following: Continued implementation leadership to guide and support adoption Education and learning program resources and infrastructure must support this as part of implementation Comprehensive stakeholder engagement and communications plan and support Local/ regional infrastructure, technology and resources to implement and support the program No specific technology solution to support peer review should be prescribed Develop communication for patients and the general public to inform them of the intent, scope and benefits of the peer review program in a way that is meaningful and appropriate to meet their information needs and support the goal of transparency overall. Further details and guidance around the implementation infrastructure will be detailed as part of the next phase of planning. 22 Expert Panel on Diagnostic Imaging Quality 2014 Health Quality Ontario