Doc No. 4. NSW Health. Independent Review of Lerner FirstNet

Transcription

1 Doc No. 4 NSW Health Independent Review of Lerner FirstNet

2 Contents Contents I Purpose of this report 3 2 Executive Summary 4 3 Introduction 14 4 Observations 21 5 Recommendations 39 Appendix A - Interview List 44 Appendix B - Acronyms 46 Appendix C - Source Documents 48 Appendix D - Chronology of Events 54 2

3 Purpose of this report 1 Purpose of this report Cerner FirstNet is the IT system selected by the NSW Department of Health to support the operation of emergency departments within the majority of public hospitals. Planning of this initiative dates back to 2002 with the first implementation commencing in The Cerner FirstNet system has now been implemented in over 50 hospitals throughout NSW, and is a core component of the broader NSW Health integrated electronic medical record ("emr") system. Following media reports in early March criticizing the NSW Health Cerner FirstNet Emergency Department ("ED") system, the Minister for Health committed to undertake an independent review of the implementation of the Cerner FirstNet Emergency Department system in NSW hospitals. The NSW Department of Health commissioned Deloitte to conduct this review of the Cerner FirstNet system and the effectiveness of its implementation. Specifically, the review has assessed the suitability of the FirstNet system to meet the clinical, functional and useability requirements of the users, and whether its continued use posed any risk to clinical safety. Disclaimer It should also be noted that the findings and recommendations presented in this report are based on information and documentation provided by representatives of NSW Health and various external stakeholders. We have not audited or independently verified that information. If the information is incorrect, incomplete or out of date (i) our report and its contents may be inaccurate and/or not meet your requirements and (ii) we will not be responsible for any loss suffered as a result of our reliance on the information. Also, while this report contains advice and recommendations, all decisions in connection with the implementation of such advice and recommendations will be the responsibility of NSW Health. 3

4 Executive Summary 2 Executive Summary.1 emr. and F In March 2000 the Report of the NSW Health Council (Menadue Report)' gave renewed focus to the implementation of a number of information management and technology initiatives to better support clinical practice at the point of patient care. In part, as a response to the Report, NSW Health submitted a Point of Care Clinical system ("PoCCS") business case in 2002, the objective of which was to obtain approved funding to implement a common State wide system, to provide an interactive clinical decision support facility to clinicians, supporting the day-to-day clinical management of their patients within the hospital setting. The business case identified Cerner as the preferred Electronic Medical Record (emr) system and a contract (head agreement) was signed between NSW Health and Cerner on 27 September Subsequent to the original contract, orders for the supply of emr modules were raised by the various Area Health Services'. The first instance of the emr, including the FirstNet3 Emergency Department (ED) component, under the head agreement was implemented at St George Hospital in the South East Sydney and Illawarra region in September Since then the emr including FirstNet has been implemented at 594 sites. The initial rollout program is due to complete by late F -:J_nd,.view A number of issues with FirstNet have been raised by ED clinicians, other hospital staff, professional bodies and academics. Issues raised include FirstNet's potential to negatively impact on efficient hospital operations and more critically, patient care. Following reports in the media in March of criticisms of the NSW Health Cerner FirstNet ED system made by Prof Jon Patrick at the University of Sydney, the Minister for Health committed to undertake an independent review of the Cerner FirstNet emergency department system implemented in NSW hospital emergency departments (The Review). In May 2011 the NSW Department of Health commissioned Deloitte to conduct an independent review of the Cerner FirstNet system and the effectiveness of its implementation. The purpose of this review is to assess the criticisms raised and advise the NSW Minister for Health and the Director-General of NSW Health on the appropriateness of continued use of the Cerner FirstNet system as a core component of the electronic medical record. ' Report of the NSW Health Council (Menadue Report), 01 March 2000, 2 When referring to events prior to 1 Jan 2011, this report uses the term `Area or Area Health Services'. Post I January 2011, the term Local Health District or District, is used. 3 FirstNet refers to the Emergency Department component of the Cerner emr product suite. It is also the name given to the NSW Health program of work for the implementation of this system across the NSW Health hospitals. Unless specifically noted, reference to FirstNet in this report refers to the NSW Health Cerner ED system and its implementation. d As at 1 June

5 Executive Summary Scope of this review The scope of this review as set out in the terms of reference was: 1) The suitability of the FirstNet system to meet the clinical, functional and useability requirements of the users, and whether it's continued use poses any risk to clinical safety 2) The effectiveness and integrity of FirstNet integration with other clinical systems, and FirstNet's capacity to ensure accurate and timely exchange of data. 3) The level of engagement, consultation and ongoing communication with ED stakeholders and users, including the effectiveness of implementation change management and end user training processes. 4) The ability of the system to support the key clinical processes of end users. 5) The response to user concerns raised through the Application Advisory Group (AAG) 6) The responsiveness of Cerner to change requests and vendor management by Health Support Services ICT (HSS) and NSW Health (the Department). 2 c exclusions The terms of reference for the review specifically excluded the following: 1) A detailed technical evaluation of the FirstNet system architecture and data model 2) The procurement processes related to the selection of FirstNet 3) The FirstNet implementation project management 4) A review of the emergency department clinical processes 2 To provide a structured framework for conducting the review a clear set of evaluation criteria across a range of dimensions was defined and then applied against an assessment of FirstNet and its implementation. The evaluation criteria, agreed to by the review's steering committee, are detailed below: Fit for purpose Whether FirstNet provides clinicians and administrators with the capabilities required to run an ED and deliver efficient and effective care. The availability and reliability of the system in supporting clinicians to ensure the highest possible quality of care can be delivered. The appropriateness of the operating environment within which FirstNet is managed and supported. The integration capabilities of the system with regard to reliability, security and adaptability to ensure accurate and timely exchange of data. 5

6 Executive Summary.5 Vendor The performance of the vendor with regard to meeting contracted performance and reporting obligations during system development, rollout and operations. The responsiveness of the vendor to requests for product changes and support..6 Change Management and Training The effectiveness of change management and communications processes to build buy-in, commitment and capacity for change. The effectiveness of the training curriculum and delivery approach in addressing the gap between current skills of clinicians and future state skills required to operate the system effectively and leverage its capability Governance The alignment of the FirstNet program with the ICT Vision, ICT Strategy and the effectiveness of governance processes with regard to monitoring and reporting on progress and decision making. 2.4 The evaluation criteria map to the Terms of Reference as detailed in the following table:. Clinical, functional & useability requirements & clinical risk Effectiveness & integrity of FirstNet integration with other clinical systems Engagement, consultation communication, change management & end user training processes Ability of the system to support the key clinical processes of end users Response to user concerns raised through the Application Advisory Group Responsiveness of Cerner to change requests & vendor management by HSS ICT & the Dept integration 6

7 Executive Summary The review commenced on Wednesday 18 May In carrying out the review Deloitte: Conducted 37 interviews across a representative range of stakeholders, as listed in Appendix A - Interview List Attended demonstrations of the system issues being experienced by some stakeholders. Four demonstrations were conducted. Obtained confidential feedback from LHDs, submitted via a confidential inbox. Over 50 submissions were received. Undertook a desktop based review of a set of documents requested from various stakeholders - refer Appendix C - Source Documents for a list of the documents examined. As a result of the work performed Deloitte made the following key observations: Fit for purpose Subject to the provision of an improved reporting capability and rectification of the faults identified in this report, the evidence suggests that FirstNet is broadly `fit for purpose' as an ED system when properly configured. Many of the criticisms of poor usability can be addressed by: Stabilising the infrastructure. Upgrading to the current FirstNet release. Adopting the most useful and proven configuration changes from sites that have already invested in useability improvements. Future planned emr project phases will reduce the issues associated with a 'hybrid" patient record thereby enhancing FirstNet's utility and value. Several ED Directors and other ED staff reported a reduction in ED efficiency as a result of the introduction of FirstNet, with one ED Director providing evidence of an approximately 20% reduction in triage performance for categories 2,3 and 4 patients following the introduction of FirstNet. Many users complained of excessive time spent in front of a data entry screen. Whilst FirstNet requires more data entry by ED staff than the systems it replaced and this can negatively impact ED performance at some hospitals, the return on the increased data input work effort will be realised in a number of ways as the full emr is progressively implemented. Problems with the FirstNet user interface should be resolved and the interface itself improved to make patient data entry and data management more efficient ' The introduction of FirstNet has changed the nature of clinical risks that exist in the ED environment. Claims of increased clinical risk arising from FirstNet's deployment, mainly ascribed to excessive staff `screen time' or an increased likelihood of mistakes must be evaluated against risk reduction achieved due to the elimination of lost or illegible patient notes, easier access to a more complete and comprehensive patient history, and reliable orders and test results. ' A 'hybrid' patient record refers to a patient record that is a combination of an electronic and paper patient record. 7

8 Executive Summary No evidence was obtained to support the claim that FirstNet has resulted in harm to a patient. FirstNet has changed work practices and has reportedly increased `screen time' for ED staff. However, time spent in capturing patient data in the ED is essential to realise the eventual benefits of an emr, and many clinicians reported to us the benefits of the electronic patient record as currently implemented. Some risk scenarios (such as mixing up labels or losing paper test results) arise from the use of a hybrid medical record but these will be progressively reduced as the patient record migrates to an electronic form. Many of the reported risks arising due to poor usability (such as the absence of composite views of certain test results, ordering tests for the wrong patient or authorisation anomalies) have been addressed by FirstNet configuration and work practice changes at some sites. These and other improvements will if more broadly implemented address many of the concerns raised by clinicians and users Sy: _ '`ins and support FirstNet system performance (response times), system availability (up-time) and the effectiveness and efficiency of support"provided to users varies considerably across the sites for the following reasons: There are significant variations in the system end-to-end application architecture (including PAS, RIS and laboratory systems), build configuration, integration approach and hardware infrastructure with distributed ownership of each component There are multiple parties involved in managing the FirstNet system and providing user support services, namely HSS, the LHDs, hospital IT services and various external vendors. The specific boundaries of responsibility of each party are not clearly defined or understood by those responsible and users. There is no consistent and coordinated approach to the provision of support and issue resolution. As a result FirstNet users are often repeatedly referred from one group to the other as they seek support, attempt to escalate performance or address access issues. Some hospitals or LHDs have invested in local additional support capabilities to help overcome these issues Integration We identified three distinct approaches for integrating FirstNet with other clinical applications with variations at different sites. This variety and complexity has added significantly to the overall management effort required to maintain and support application integration. We also noted that the responsibility for interface management varies and is often not well defined. As a result, performance and reliability of interfaces varies widely across sites. In addition, at some sites FirstNet went live without a number of core interfaces being implemented resulting in a need for workarounds - for example, ED staff manually printing orders to pass to pathology staff for re-keying into the pathology system. Integration relies on well defined messages successfully passing between systems. Public criticism of FirstNet suggested that not all interfaces properly supported all the required messaging. Our review determined that in the majority of cases this was associated with integration to the PAS. There are no current reports of message loss or adverse clinical outcomes as a direct result of integration issues. Cerner's use of HL7 for message definitions and SNOMED for message content appears to be generally well applied, although the support of a restricted set of HL7 messages requires workarounds. In addition, system users are provided limited visibility of messages that contain errors or have failed. While this is a weakness, it does not appear to have had any adverse clinical consequences The scope of Cerner's responsibility is comprehensively established via three related contracting mechanisms (the original head agreement IT-135 dated 27 September 2002, GITC Official Orders, and subsequent Change Requests). While the Cerner relationship has not been formally managed by HSS 8

9 Executive Summary as specified in these agreements, our review concludes that HSS facilitated an effective and productive working relationship with the vendor. No significant performance or delivery gaps by Cerner were identified.?.6.6 B The implementation approach adopted by the Department was that of a sequential rolling model whereby a team would implement the system, provide training and cutover support, before moving onto the next site. The approach and templates (base documents) for change management, communications and training were developed centrally by HSS. Each of the LHDs then localised the approach and templates to suit their specific needs and were responsible for delivery. While clinician feedback on capability and quality of work of the implementation teams was consistently good, with positive comment on the team's knowledge of the system, clinical processes and implementation practices, a number of factors negatively impacted the effectiveness of the change management and training delivered as part of the site implementation process: At some sites the size of the on-site implementation team and the team's duration of stay was not sufficient The implementation approach did not allow for follow up reinforcement of training The impact on local work practices was not always properly addressed. Evidence was provided to our review of inconsistent support by senior management at some sites for the implementation process, specifically in ensuring training attendance and implementation of changes to local work practices. As a result, clinicians at some sites were not adequately prepared for using the system and the changes to clinical work practices imposed by the system (for example, 'clerking' before triage) Gow Governance applies to both the program established for the implementation of FirstNet and the operation and use of the FirstNet system itself. This review identified the following issues which negatively impacted the effective governance of FirstNet: FirstNet is a core component of the broader EMR program. A high-level planning roadmap for the program and other initiatives is maintained by NSW Health for the purposes of planning business cases and budget submissions over a five year period. The program lacks a more formally documented vision for the use of technology to enable clinical practices. The effectiveness of the Application Advisory Group (AAG), a core component of the FirstNet governance model has eroded significantly over time as clinician participation in this forum has declined. With some exception, FirstNet reporting is inadequate for effective governance of ED operations At some sites users reported shared user sessions to avoid delays associated with logging out and then logging back in as a new user. The requirement for clinicians to enter a user identifying pin number to authorise a transaction has been disabled at some sites. This means that it is not possible to reliably determine the identity of the clinician processing transactions. This issue is exacerbated where clinicians share computers or user sessions. 9

10 Executive Summary S f 1 ' ins T F 'd c, i i1 4 and D_ ANet initiate a r af,tion p --m to address in this repor NSW Health has implemented FirstNet as part of the overall emr program to 59 hospitals across New South Wales. While there have been a number of reported issues associated with the deployment, use and operation of the system, many clinicians have reported that it now adds considerable value to the delivery of care and the operation of their Emergency Departments. Many of the issues that have been reported and observed relate to system operations, governance, implementation, leadership and training, and not specifically to the capability of the FirstNet system. While this review confirmed that there are a number of issues with the use, implementation, support and configuration of FirstNet, we believe these can be remediated via an appropriate program of work Rec- cl t' _ Define an(' ' iiplerent site specific - " 1 plans to nuance and t.~. ` 1 % ssdes.ng *rstnet it-1 A range of issues and user frustrations were identified across the FirstNet sites we examined. These are impacting user satisfaction and the effective use of the system, highlighting the urgent need for a formal program of site specific remediation activities. These remediation activities include remedial training, configuration and in some cases, upgrades to the currently implemented system. A well defined plan and program. of work needs to be put in place to bring all sites to a base level of acceptable infrastructure, functionality, useability and user training. We further recommend that this remediation program be delivered by resources other than those already responsible for the current implementations or the planned emr phase 2 to avoid resource conflicts with these activities. o standardisatio;, ;- nd eff c e-ncie, c While NSW Health has adopted the concept of a State Base Build (SBB) - many of the implementations are deployed on different operating system versions; different Cerner versions; different SBB versions; different local configurations; and in some cases, local additions that are outside of the scope of the emr program. This is driving inefficiency and complexity in the use, operation, support and management of these systems. NSW Health should expand the scope of the SBB to include elements added in local implementations, and then standardise where appropriate the use of this expanded SBB across the LHDs. This will improve system operation and support efficiency and share the benefits derived through successful local configuration changes across all FirstNet users. The current plans for the implementation of EMR phase 2 and medications management capability should build on the SBB model proposed above. Reporting warrants a specific remediation focus given the high level of frustration expressed during the user interviews and should be considered for special attention when upgrading the SBB. NSW Health has made substantial steps toward the implementation of an emr for all patients presenting at hospitals across the State. The emr will provide a basis for participating in the broader national 10

11 Executive Summary Personally Controlled Electronic Health Record (PCEHR) program, while providing the longitudinal record of patient care essential for addressing the ever increasing demands on the health system. As IT systems are increasingly used to support the delivery and improvement of health care, it is important that senior clinical input drive the vision and requirements for the use and future development of clinical systems. We recommend the creation of a Chief Medical Information Officer (CMIO) in NSW Health to provide this direction. A primary responsibility of this senior executive will be the development of a vision for the capabilities necessary to enable the care delivery strategies of the Department. This vision should provide the basis for all systems implementation strategies and will be fundamental to the successful use of existing and future clinical systems ' future funding requests i 1 W provisions for both the inith c: tion as weii as the on-going c.,,, cost The review identified that the funding model for the FirstNet program has been very `project' driven. While this approach has successfully supported the initial implementation of the system, it has not expressly provided for the iterative or retrospective application of enhancements resulting from lessons learned and the ongoing clinical process improvement that typically takes place in an ED. No funding is provided under the current funding model for on-going system development and training support after the initial implementation. This has a negative impact on user satisfaction and adoption and the harvesting of benefits expected from the system. It is therefore important that funding for the future phases of the emr program includes both implementation and on-going operating costs. 2.7.E Tin: AsseE go gee and organisational structures y ofciinicl. There are multiple parties involved in the delivery of FirstNet and the broader emr program, including HSS, Local Health Districts, Cerner, other vendors and the Department. The specific responsibilities of each organisation are not clearly defined or understood by all stakeholders. This uncertainty has driven inefficiency and stakeholder discontent. An improvement to the governance and organisational structures required to effectively deliver a computer enabled clinical care system within NSW Health is needed. Any changes to organisation structure, delivery and support model, and governance processes must be clearly defined and implemented R - c Enhar i te, to The success of FirstNet and the broader emr program requires highly reliable system availability and responsive user support. This is currently not the case at many sites, resulting in significant user frustration and an inefficient use of valuable resources. It is important that the Department undertake a review of the existing system operation and user support resources and processes with the aim of establishing a new transparent and effective system and user support capability that is appropriate for clinical applications. As the hospitals across the state move more towards a paperless care delivery model, the reliance on the availability of the clinical systems and the patient information in those systems becomes critical to a clinician's ability to deliver effective care. It is essential that highly available and high performance I1

12 Executive Summary infrastructure be provided to support the delivery of this critical IT service. The review established that the existing IT infrastructure for FirstNet does not universally meet this minimum standard. Additionally, the responsibilities of the multiple parties involved in delivering support are not clearly defined. A review should be undertaken of the infrastructure delivery approach, and an assessment made as to the suitability and capability of each contributing party. This review should also include an assessment of alternate sourcing approaches. Many international users of Cerner do not run their own data centres, and outsource the hosting and management of the core infrastructure required to run the application to specialist third parties. 12

13 Executive Summary The recommendations map to the Terms of Reference as follows: Clinical, functional & useability requirements & clinical risk Effectiveness & integrity of FirstNet integration with other clinical systems Engagement, consultation communication, change management & end user training processes Ability of the system to support the key clinical processes of end users Response to user concerns raised through the Application Advisory Group Responsiveness of Cerner to change requests & vendor management by HSS ICT & the Dept etain FirstNet ite specific emediation plans Expand & onsolidate BB Funding model review linical program overnance review upport model review nfrastructure "livery pproach review 13

14 Introduction 3 Introduction 3.1. I E V Li. to t he NSW Health e MR program and FirstNet The FirstNet` emergency department system is part of the NSW Health emr journey which commenced some 10 years ago. In March 2000 the Report of the NSW Health Council (Menadue Report)' gave renewed focus to information management and technology initiatives that better support clinical practice at the point of patient care. As part of the response to the Report in 2002 NSW Health submitted a Point of Care Clinical (PoCCS) business case the objective of which was to obtain approved funding to implement a common State wide system to provide an interactive clinical decision support facility to assist clinicians in the day-to-day clinical management of their patients within the hospital setting. The business case identified Cerner as the preferred Electronic Medical Record (emr) solution and a contract (head agreement) was signed between NSW Health and Cerner on 27 September This contract forms the basis of the State's relationship with Cerner. PoCCS, which was later renamed the Electronic Medical Record (emr), was identified as a key component of NSW Health's Information Management and Technology Strategy. NSW Treasury approved, but only partially funded, the PoCCS business case in Following approval, limited progress was made in implementing the strategy. At the end of 2005 a tender was released to explore options for an alternative emr solution. The evaluation confirmed the Cerner Millennium" suite as the appropriate solution for the NSW Health emr program. Additional funding was sought and approved to significantly accelerate the implementation of components of the emr including Results Reporting, Order Management, Electronic Discharge Referral, Operating Rooms and Enterprise Scheduling, and Emergency Department (FirstNet). Following the signing of the original contract, "Official Orders" for the supply of emr component modules were entered into by the Local Health Districts (LHDs). The first instance of the emr including FirstNet for emergency department management and tracking was implemented under this agreement at St George Hospital in the South East Sydney and Illawarra region in September Since then FirstNet has been implemented at 59 sites and the rollout is due for completion by late FirstNet refers to the Emergency Department component of the Cerner emr product suite. It is also the name given to the NSW Health program of work for the implementation of this system across the NSW Health hospitals. Unless specifically noted, reference to FirstNet in this report refers to the NSW Health Cerner ED system and its implementation. Report of the NSW Health Council (Menadue Report), 01 March 2000, 8 Cerner MillenniurTM is a comprehensive suite of solutions supporting personal and community health management 14

15 Introduction For the month March 2011' the following system use data was reported. Total transactions Total Orders Average Daily Users Average Daily orders 86m 2.84m 17,188 91,741 x The State Base Build (SBB) for the emr is a framework of content and system configuration parameters which has been used as the baseline for NSW Health implementations of the Cerner Millennium suite of products. The St George Hospital site implementation formed the basis of the SBB. The development of the current SBB has been an ongoing process and the SBB continues to be developed. Approved change requests have been incorporated into subsequent versions of the SBB. Sites are required to use the SBB as the basis for their implementation, but are allowed flexibility in the way the system is configured at the site level. In practice this flexibility, coupled with the implementation of different versions of the SBB has resulted in significant variation in configurations across the State, LHDs and sites. This variation is one of the drivers behind the different levels of user satisfaction with FirstNet. There are currently 4 versions of the SBB implemented across the 8 former Local Area Heath services (or 15 Local Area Health Districts). 3ound A number of issues with FirstNet have been raised by ED clinicians, other hospital staff, professional bodies and academics. Issues raised include FirstNet's potential to negatively impact on efficient hospital operations and more critically, patient care. Following reports in the media in March of criticisms of the NSW Health Cerner FirstNet ED system made by Prof Jon Patrick at the University of Sydney, the Minister for Health committed to undertake an independent review of the Cerner FirstNet emergency department system implemented in NSW hospital emergency departments (The Review). In May 2011 the NSW Department of Health commissioned Deloitte to conduct an independent review of the Cerner FirstNet system and the effectiveness of its implementation. The purpose of this review is to assess the criticisms raised and advise the NSW Minister for Health and the Director-General of NSW Health on the appropriateness of continued use of the Cerner FirstNet system as a core component of the electronic medical record. The terms of reference for the review are as follows: 1) The suitability of the FirstNet system to meet the clinical, functional and useability requirements of the users, and whether it's continued use poses any risk to clinical safety 2) The effectiveness and integrity of FirstNet integration with other clinical systems, and FirstNet's capacity to ensure accurate and timely exchange of data. Lights On Report for March

16 Introduction 3) The level of engagement, consultation and ongoing communication with ED stakeholders and users, including the effectiveness of implementation change management and end user training processes. 4) The ability of the system to support the key clinical processes of end users. 5) The response to user concerns raised through the Application Advisory Group (AAG) 6) The responsiveness of Cerner to change requests and vendor management by Health Support Services ICT (HSS) and the NSW Health Department. The Review's scope specifically excluded an examination of the following: 1) A detailed technical evaluation of the FirstNet system architecture and data model 2) The procurement processes related to the selection of FirstNet 3) The FirstNet implementation project management 4) A review of the emergency department clinical processes. To provide a structured framework for conducting the review a clear set of evaluation criteria across a range of dimensions were defined and then applied against an assessment of FirstNet and its implementation. The evaluation criteria, agreed to by the review's steering committee, are detailed below: Evaluation Criteria Fit for purpose Description Useability Is the user interface appropriate to support the operational, time and care delivery demands of an ED clinician? Does the system provide a user interface that is appropriate for each of the administration and care provider staff using FirstNet? Accessibility Does the system provide an effective and secure mechanism for clinicians to access the required information and services? Does the system provide access to information through multiple and appropriate devices (e.g. mobile, touch screen, real-time monitoring)? Functional Fit Does the system provide appropriate support for all of the functions required to both manage an ED and deliver effective patient care? Does the system effectively support other reporting, administration and care delivery processes within a hospital? Clinical Process Support Can the system be flexibly configured to support the local work practices at each location? Does the system enable the rapid implementation of new processes and models of care? 16

17 Introduction Evaluation Criteria Clinical Risk Description Information Integrity Is the information provided by FirstNet always reliable, and do the clinicians have confidence in the information provided? Is information reliably transferred between systems? System Responsiveness Does the system provide information in an adequately responsive timeframe? Identity and Access Management Does the system facilitate user identification and enforce user tracking of all transactions entered? Information and System Availability System Operation and Support Is the system availability appropriate for clinical environments? Service Delivery and Support Is the FirstNet and supporting technology environment appropriately managed to meet the availability, capacity, access and performance requirements of the hospitals? Are users of the system well properly supported when raising requests for support or system changes? Infrastructure Management Are appropriate strategies and plans in place to ensure that the technology infrastructure adequately supports the current and future operating requirements of FirstNet? Application Management Are appropriate processes in place to ensure the effective planning, development, implementation, maintenance and support of the FirstNet application? Integration Are appropriate application change management and configuration management processes in place? Integration Capability Is the integration between FirstNet and other systems reliable? Is the integration architecture sufficiently open and adaptable? Standards and Compliance Are the system messages compliant with agreed standards? Does the system standards? Error Handling use code-sets that are compliant with agreed Does the system provide adequate detection, logging and notification of errors and exceptions? Does the system facilitate appropriate response and rectification of errors? 17

18 Introduction Evaluation Criteria Vendor Management & Support Change Management and Training Description Contract and Vendor Relationship Management Has the vendor met contracted performance and reporting obligations? System Support Effectiveness Has Cerner responded to requests for product changes and support in a sufficiently timely and satisfactory manner? Change management and communications Do management processes exist to effectively assess the support required by stakeholder groups, identify potential risks and barriers and design strategies to address these and build buy-in, commitment and capacity for change? Are stakeholders communicated with effectively to inform them of the system changes, foster buy-in and enable feedback mechanisms to enable dialogue with users? Training effectiveness Does the training curriculum and training delivery approach address the gap between current skills of clinicians and future skills required to use the system effectively? Governance ICT Vision and Strategy Does FirstNet align with the ICT Vision and Strategy? Business Operations Governance Does the system enable effective reporting to monitor performance of the organisation? Does the system adequately support the operational and clinical governance processes within the hospitals? System Governance Are the processes for managing the FirstNet implementation program effective? 18

19 Introduction The evaluation criteria described above can be cross referenced to the Terms of Reference as follows: Clinical, functional & useability requirements & clinical risk Effectiveness & integrity of FirstNet integration with other clinical systems Engagement, consultation communication,change management & end user training processes Ability of the system to support the key clinical processes of end users Response to user concerns raised through the Application Advisory Group Responsiveness of Cerner to change requests & vendor management by HSS ICT & the Dept Fit for purpose linical Risk ystem,perations & upport Integration endor Management & Support hange anagement & raining overnance 19

20 Introduction The review commenced on Wednesday 18 May In carrying out our review Deloitte: Conducted 37 interviews across a representative range of stakeholders, as listed in Appendix A - Interview List Attended demonstrations of the system issues being experienced by some stakeholders. Four system demonstrations were conducted. Obtained confidential feedback from LHDs, submitted via a confidential inbox. Over 50 submissions were received. Undertook a desktop based review of a set of documents requested from various stakeholders - refer Appendix C - Source Documents for a list of the documents examined. 20

21 Observations 4 Observations This section details the observations gathered during the review, categorised by evaluation criteria 4`Fit for purpose' is a characteristic of product quality. Quality is not an absolute characteristic of a product but is determined by the product's use. A product can be said to be `fit for purpose' if it is sufficiently suitable for the intended purpose or use. Themes for this criterion that are common to multiple sites and accounts fall into four groups: usability, accessibility, functional fit and clinical process support Usability Usability can be defined as the ease of use and `learn-ability' of a product. FirstNet presents to users via a 'thick client' window and requires both keyboard and mouse interaction for data entry. The user interface broadly employs familiar Microsoft Windows user interface (UI) conventions but exhibits some inconsistencies between views and data entry forms. Basic activities such as searching and viewing patient details, status checking and status monitoring, and navigating between functions typically requires explicit mouse-clicks, keystrokes, and usually a combination of both. User interface is generally considered non-intuitive and difficult to use Many users reported that they did not find the FirstNet user interface useable or intuitive, based on its screen and menu design, and the way commonplace activities are supported. Users consistently raised the following criticisms: Some screens are crowded and present unrelated or irrelevant information or fields The patient tracking screen was consistently criticised for presenting too much information There are too many icons for even regular users to remember, and the icons themselves are ambiguous Many basic and frequently used actions take too many clicks to complete The same key provides different functions in different fields or screens The application is inconsistent with basic Windows conventions (for example, the Enter key cannot consistently be used as a substitute for completing a form or action) The application does not support keyboard alternatives to mouse-clicks, which necessitates continuous alternating between the keyboard and mouse to navigate or complete a form. Some of these usability criticisms have been addressed to varying degrees through local changing of configuration parameters, both at the user and product level. For example, user-defined patient lists can be created by users, with each list presenting on its own separate tab. If the number and names of patient lists is not constrained or made consistent, a large number of ambiguous tabs appear on the FirstNet landing screen. In addition, a number of small but significant usability improvements have been made at some sites by Cerner. Two examples are more readable discharge summaries and the addition of an asterisk in the patient row on the `All Patients' table to indicate a completed theatre booking. The design of some high-use screens negatively impacts work efficiency A number of ED directors commented that the introduction of FirstNet had negatively impacted ED productivity, necessitating the scheduling of additional resources to compensate. One ED Director provided evidence to substantiate this claim. 21

22 Observations Examples of issues relating to screen design that contributes to lowered work efficiency raised during the interviews include: Users spend too much time navigating around multiple poorly designed screens. For example, the `All patients' table (one of the most frequently used screens) must be scrolled horizontally and vertically to access rows (patients) and columns (data on each patient). While some reported this as merely inconvenient and time-wasting, others claimed that it represented a risk as it was possible to click on the wrong patient's row to launch a form to order tests. It should be noted that this specific issue can be fixed through local configuration as was successfully done at one site by freezing the first two columns of the screen containing the patient number and name so that the patient's identity remained visible as the remainder of the window scrolled. The same information sometimes needs to be entered multiple times when ordering tests. The system also prompts users for inappropriate or irrelevant information, or information which wastes time. There are reports of anomalous or apparently incorrect FirstNet behaviour, such as loss of comment data from the `Pre-Arrival' form when the 'Arrival source' drop-down selection is changed. (It was pointed out that the lost data can be recovered by simultaneously pressing the Control and Enter keys). That said, some sites do not use the FirstNet `Pre-Arrival' screen while others do not consider the potential for loss of this data to be important. Configuration of certain labels and codes is inconsistent across the sites The decision to adopt the SNOMED taxonomy was made as part of the transition to FirstNet. At some of the earlier implementation sites SNOMED code-sets were not edited or reduced to meet local requirements or practice. This resulted in practitioners being frustrated by the large number of irrelevant codes and unfamiliar (predominantly American) terminology in code names and labels. Some sites have addressed this through local configuration to reduce the code set size - by removing irrelevant codes, or in one case by defining 'Favourites' or categories of frequently used codes. A similar situation exists for orders. Users agreed that the top 20 orders account for approximately 98% of all orders placed. One site implemented the `top 20 orders' as 'favorites', however no mechanism exists to share this configuration with other sites. Visibility of the patient record varies across sites A clinician's visibility of a patient's medical record in FirstNet varies across sites, depending on the progress of the emr program implementation at that site. At hospitals where Cerner PowerChart is not used or is not widely used, a patient's medical record will include both paper and electronic forms". At some sites, the handover from the ED to a ward involves printing the electronic ED records to create the complete patient record for the ward. Clinicians at hospitals where PowerChart is used in both the ED and the wards are often strong advocates of the electronic medical record, citing significant benefits relating to the visibility to patient records both within and between episodes at their hospital and within their area over time. Other reported benefits include time savings from not having to search for paper-based notes, the elimination of lost and unreadable notes, and being able to always see patient arrival and discharge times as well as the time they were seen by a doctor. '0 Referred to as a 'hybrid' patient record. 22

23 Observations System response times are variable and depend on location and time of day System responsiveness is perceived as a usability issue. The review identified a number of sites where users reported unacceptable and frustrating response times. Clinicians do not distinguish between FirstNet and the infrastructure or hosting services, so a problem with Citrix or a network connection is perceived as a problem with FirstNet. Unreliable or poor system responsiveness erodes confidence in FirstNet and the emr more generally and is viewed as a serious and unacceptable impediment to work by ED staff Reports of system response times vary widely by area. Some practitioners report system responsiveness that is consistently slow to the point of being unusable. Comments such as the system is painfully slow three out of seven shifts' and `screens are slow to refresh' were common from these sites. Reports of EDs reverting to whiteboards for patient tracking during periods of unacceptable response times or unscheduled system downtime were made by several sites. It was also claimed that in many cases information collected on whiteboards was later never entered into FirstNet or captured in other patient files. Conversely, other sites (and areas) reported system responsiveness as being acceptable and relatively consistent. The differences are likely to be a function of hosting and infrastructure capacity and reliability, particularly the Citrix infrastructure capacity and network bandwidth Accessibility Accessibility refers to the ability of ED staff to freely access and use FirstNet during their shifts. Accessibility is a function of the number, physical placement and hardware configuration of workstations, as well as the design of the FirstNet user interface. Too few computers in EDs It was claimed that the number, position and configuration of computers in EDs is not always conducive to effective work practices. In most cases, clinicians and ED staff `share' a computer, sometimes creating contentions during busy periods. ED staff may or may not share a user session - at some sites doctors and nurses share a logged-on user session for viewing results and doctors only change users when electronically signing an order or discharge. Some clinicians stated that effective use of FirstNet requires large monitors and that these are either not installed or there are too few. Logging in and changing users for authorisations While most clinicians reported acceptable logon times at their hospitals, a limited number of clinicians reported frustration at having to wait for up to 3 minutes to logon to FirstNet. The logon process can take up to 6 mouse-clicks and presents unnecessary confirmation and `splash' screens. Improvements (such as a single-click logon desktop icon) have been implemented more recently at some sites. FirstNet supports the option for enforcement of an authorising user (a doctor) to electronically sign an order or discharge summary. This ensures the appropriate authorisations are captured against patient records and orders. At sites where this feature is enabled, a doctor must log in or change user (by entering their username and password) to 'sign' and complete an order or discharge. At some sites, this feature is not turned on, and some clinicians reported that doctors had submitted orders and discharges on another staff member's session. System availability varies significantly across sites System availability is affected by planned and unplanned outages, where an outage is defined as the unavailability of the system or any of its essential parts for normal use. On planned outages, several sites reported the system initially being taken down for periodic maintenance or upgrades with little regard for the impact on EDs (for example, on Saturday nights when EDs are busy). These same hospitals reported recent improvements with consultation and agreement on the times and the likely duration of planned outages. 23

24 Observations On unplanned outages, site representatives gave very different accounts. Several sites reported an unacceptable degree of system unreliability for the first 12 months after cutover, as well as unacceptably long downtimes when these unplanned outages occurred. In most cases, improvements in availability and restoration times have been made over the last 6 to 12 months. Another accessibility issue arises where FirstNet prevents access to a patient record that is current (i.e. not discharged) at another hospital in the area. ED staff work around the problem by either calling the other hospital to request a staff member to discharge the patient, or they create a duplicate patient record and merge the records at a later time. 4. L3 Functional fit Functional fit refers to the degree to which FirstNet provides the right functions to support the activities and workflows of ED staff. Reporting capability and access to data is inadequate Reporting, access to ED data and ad hoc (or on-demand) reporting was widely criticised. The criticisms fall into three categories - too few reports were implemented; FirstNet's support for one-off access to ED data is inadequate; and data quality in extracts or reports is often poor. The criticism that too few reports were implemented in FirstNet derives from a comparison with its predecessor EDIS, which provided reports which were not replaced in FirstNet, particularly reports on the numbers of cases, services or diagnoses in a given period. This is a source of significant frustration for some clinicians and has created additional work for data managers at some sites. On the system's support for access to ED data and ad hoc reporting, a number of ED Directors reported significant difficulties in getting access to one-off queries or reports providing data over a period of time to support departmental management, research or trend analysis. This kind of reporting is distinct from FirstNet's support for user-defined lists of current ED patients. The combination of weak reporting capability and lack of visibility of the data within the system leads clinicians to think `there is far more data going into the system but we are able to get far less out'. On poor data quality, some sites reported little confidence in the integrity of FirstNet reports. One ED Director provided a sample report reflecting a total of only 25 patients having visited the ED on a given day when the actual number was known to be 125. red to a lost not,-,, ED Director ED performance data requires manual remediation NSW Health's `Demand and Performance Evaluation Branch' routinely collects data from the State's EDs for a range of purposes including performance management, customer satisfaction, cost effectiveness, capacity, workforce and service planning and health research. Since the commencement of the initial FirstNet rollouts, the Demand and Performance Evaluation Branch has reported a significant increase in data that fails basic validation criteria. Examples are missing data or poor data quality including missing fields (such as treatment time, preferred language and insurance status), treatment time before triage time, and departure before triage time. The Demand and Performance Evaluation Branch has implemented workarounds to partially remediate FirstNet data requiring manual intervention by both the hospitals and the Branch. Despite these workarounds, the integrity of some data used by the Branch continues to be compromised. Poor discharge summaries Inadequate discharge summary capability is one of the most common criticisms raised against FirstNet. The standard templates provided are clumsy and the editing and formatting capability provided is limited. Some ED doctors reported being 'embarrassed' by the quality of the discharge summaries produced by FirstNet, resorting to external word processors to manually format the report and then copying the 24

25 Observations formatted document back into FirstNet. Some sites reported having developed local templates and formatting and in this way have addressed this issue. Functionality is generally adequate Apart from the usability problems described above, clinicians were broadly satisfied with the functional capabilities provided by FirstNet, with the noted exception of reporting. Benefits cited by a number of clinicians include the value of electronic records and notes, increased transparency of patient flow through the ED, longitudinal visibility of patients over multiple visits both at the clinician's hospital and to other hospitals in the area, integrated ordering and results, triage tracking and support, and support for improved decision-making Clinical process support Clinical process support refers to the degree to which FirstNet supports the clinical activities and processes of the ED. System introduces discipline around processes Clinicians reported the need for changes to work practices following the implementation of FirstNet. While some appreciated the enhanced enforcement of disciplines, others cited unwelcome disruption of local practices. One site reported re-numbering their ED rooms because they found it easier to do than changing the room designators in FirstNet. A significant process change introduced by FirstNet involves `clerking' patients (which includes establishing their identity in the PAS) before triaging. The need to access the PAS to establish the patient's identity before triage was cited as an `unnecessary administrative burden on clinical staff and a contributor to increased clinical risk'. One hospital claimed it could take up to 10 minutes to get a patient record from the hospital's PAS while others described it as a 'minor adjustment of work practices' that did not significantly impact staff or increase risk because it simply required re-sequencing of existing tasks and was easily accomplished by accessing the PAS via a single button added to the screen. Insufficient user support for local variations in clinical processes Many ED clinicians criticised FirstNet for its apparent rigidity and enforcement of uniform clinical processes. Some of this criticism derives from counter-intuitive aspects of the user interface and other usability concerns discussed earlier. Users also stated that FirstNet introduced too much clinical process change at one time and that post-implementation support for users was insufficient to ease the transition. Users reported the need for greater support pre and post go-live in order to properly understand their options to customise FirstNet, both at a user and site level. A lack of user awareness of local configuration options was evidenced, especially at sites with limited local user support capability. This includes aspects such as user-defined patient lists, customised diagnostic and allergy code sets, changing American terminology, and the use of `favourites'. 4.2 Allegations have been made that the introduction of FirstNet has significantly raised clinical risk for patients. During the review, we assessed aspects of system functionality, performance and integrity that presented the potential to contribute to a change in the nature of clinical risk. Our assessment is presented using the following themes: system data and integration integrity, clinical process change, data quality, system availability, and enforcement of authorisations. Lack of system data and integration integrity increases clinical risk It has been claimed that the system lacks data integrity and as a result has been known to `lose' patient records or data. There are two potential scenarios leading to loss of data - loss of a test order or results 25

26 Observations and loss of a patient record while in the ED. In general, integration'' was reported as reliable, notwithstanding some initial field-level mapping anomalies which have been subsequently corrected. No evidence was found of repeatable or systematic message loss at interfaces. Similarly, no evidence was found to support concerns about the loss of an ED patient's record from FirstNet. It appears that it is possible for a patient's record to become assigned to a clinician without that clinician being aware or present and this scenario may have been interpreted as the loss of a patient's record. The patient would, however, still appear on the `All Patients' list. Low data quality increases clinical risk It has been claimed that the introduction of FirstNet compromised ED performance data collected by NSW Health. This is because field validation in FirstNet is currently turned off in the State Base Build'', so it is possible for incorrect dates, codes and other structured data to be saved or not entered at all. It is also possible to save inaccurate treatment start and end times. Poor data quality is having significant downstream consequences and this problem needs to be addressed. However, because performance data is not used by clinicians to directly support patient care, this deterioration of data quality cannot be claimed to have significantly increased clinical risk System availability and responsiveness Insufficient system availability and responsiveness increases clinical risk An ED system must have high availability. There are many accounts of unacceptable or unscheduled FirstNet downtime at some sites, although in most cases reports indicate that this has improved over time. When FirstNet is unavailable, the ED loses visibility of its patients, their assignments, statuses, orders and results. Typically, ED clinicians revert to a whiteboard for basic patient tracking. A fallback and subsequent recovery from a period of unavailability arguably introduces some clinical risk on the basis that information must be reconstituted on a whiteboard and then transferred back to FirstNet when it becomes available Identity and access t Lack of enforcement of authorisation increases clinical risk The system requirement for the authorisation of orders, diagnoses, test results and discharges by a doctor is disabled at some sites, presumably because the task of changing users is cumbersome and slow. As a consequence, any ED staff member at these sites can order or authorise tests or discharge a patient from any `logged-in' computer. Some interviewees reported that it is also possible for a patient to be discharged without all assigned staff having completed their notes and that a staff member can sign off an ECG on the current (potentially different) user's login. This compromises the integrity of authorisation processes and the ability to audit information in the system. Authorisations should be enforced and made to work effectively to mitigate this risk. The future introduction of Medications Management will further raise the importance of the capability to record and enforce authorisations for prescribing and medicine administration. 4.2._ Changes to clinical processes increase clinical risk Some ED Directors claimed that clinical process changes forced by FirstNet (primarily data entry and `clerking' before triaging) have raised clinical risk due to the time taken for patient lookup and admission during the triage process. Other ED Directors stated that risk has not increased because patient lookup " Refer section 4.4 for detailed observations on the integration between FirstNet and interfacing systems (primarily PAS, pathology and radiology). z This was presumably done to remove a clinician's obligation to `fix' erroneous data when handling an emergency situation. 26

27 Observations or administration is handled separately from triage, and patient admittance processes allow judgement to be exercised at all times. At most sites, FirstNet allows a patient to be entered by ED staff as unidentified (with name and basic identifying fields) and then linked to the PAS patient record at a later time. Confusing UI design increases clinical risk A number of clinicians and respondents argued that poor user interface design makes interactions with FirstNet unnecessarily time consuming, and as a result ED staff spent more time doing data entry than caring for patients, thereby decreasing productivity and increasing clinical risk. The following usability issues were identified by interviewees as potential causes of increased clinical risk: Note-taking in FirstNet is considered by some ED staff as so cumbersome that they do not bother, resulting in reduced clinical documentation Navigation is found by many staff as counter-intuitive and multi-layered to the point where it slows down their work Searching and code selection problems make selecting diagnostic, allergy and test codes time-consuming, particularly the SNOMED diagnosis code set which is considered by many to be too large, difficult to navigate and cluttered with irrelevant codes13 and obscure names Tests may be ordered for the wrong patient as a result of the patient name disappearing when scrolling in the patient tracking screen Some test specifications are poor (for example, it is reported that it is possible to order an X- ray or image without specifying the part of the body to be imaged) The way that FirstNet reportedly presents certain test results could obscure recognition of a potentially life-threatening condition. While these usability issues can potentially introduce clinical risk, it should be noted that some of these issues have already been partially or fully addressed at some of the sites that have carried out local configuration Service delivery ar. - rt Responsibility for service delivery and support of the FirstNet system is spread across multiple organisations: With the exception of non SBB sites, HSS is responsible for management and support of the core FirstNet system including the hardware and delivery services. HSS also manage the development and maintenance of the SBB and the associated governance process. HSS provide a first level telephone support service to record and respond to FirstNet service requests. Local health services provide local training, support and configuration services, as well as the first level of governance for changes to the State Base Build. Cerner is responsible for FirstNet third level support and the development and implementation of software changes based on direction from both the Application Advisory Group (AAG) and local area or site requests in some cases. Clarity of process for management of system support varies by site and area There was evidence of considerable variance in users' understanding of issue resolution and support processes across the areas and sites. Given the number of parties involved in support roles and the varying investment in local support, the processes for an individual to seek help with a system issue and receive follow-up resolutions and closure are not always clear. 1 3 Examples include `San Miguel Sea Lion virus' and `Mexican hairless doallergy'. g 27

28 Observations Responsiveness and effectiveness of system support varies by site and area The varying effectiveness of local and centralised support is driving considerable discontent in a number of sites. The rolling implementation team approach did not leave every hospital with an equivalent level of system support capability. Each site has responded differently with some sites recruiting dedicated local support resources, some training `super-users', some relying on scarce regional support, and others relying on the HSS provided centralised support. This has in turn led to considerable variance in user skills, with some clinicians being relatively expert and able to establish specific lists and shortcuts, while others have only basic FirstNet skills. The communication and management of planned and unplanned outages varies by site and area While planned outages were reported to be generally adequately communicated, their timing was often a cause of frustration. The incidence of unplanned outages appears to vary significantly by site and area. Some sites report that system performance (i.e. response time) often deteriorates to the point where the system freezes completely. When this occurs, feedback on expected resolution timeframes is often minimal. For example, some users reported being given no direction other than being told to `keep on trying until you can log in'. Change request process lacks transparency and can be unacceptably lengthy While the change request processes are reasonably well defined, the visibility of the status of requests is often inadequate. Many users stated that when requests were submitted for review there was often no feedback on status, and in some cases change requests took a year or more to be processed Infrastructure management There are multiple hardware and software components required for the delivery of end -to-end IT support of an ED. At most hospitals FirstNet integrates with a local PAS, RIS, PACS, and laboratory systems. This integration is supported by local networks and in some cases local integration engines (e.g. egate, JCAPS), so that the centralised integration infrastructure provided by HSS may integrate directly to local systems, or with a local integration infrastructure including other systems. Also, in some areas there are multiple implementations of each type of local system that are integrated to a single Cerner instance. This infrastructure complexity, design variance, and distributed component ownership presents a difficult infrastructure management scenario. Infrastructure is managed by various parties (HSS, area and /or site) HSS has responsibility for managing the Cerner environments and the integration infrastructure required to interface to Cerner. The local areas are responsible for the delivery and management of local systems and integration with these systems, to enable a complete business or clinical process to be supported. While management responsibilities are well defined, there is no apparent single point of accountability for end-to-end system operation, which makes it difficult to assess system problems, and diagnose what remediation is required. Various levels of resource and capability available by site We observed considerable variances in system performance across implementations. At some sites the system performs well with screen refresh times being well within what most users consider acceptable. At other sites it was reported that the system often becomes so slow that it becomes unusable. For example, one site reported that during the initial stages of FirstNet, there were periods when the system regularly `almost stopped' driving clinicians to revert to manual processes for an hour or two until performance gradually improved. This was reported as occurring approximately three days in every seven day roster. Non-SBB sites generally reported a consistently higher level of support satisfaction, as did sites that appear to have generally benefitted from higher levels of investment to ensure the successful introduction of FirstNet and the broader emr program. 28

29 Observations Effective use of FirstNet requires access to appropriately specified and located PCs and monitors A number of Cerner screens present information that is wider than the display width of a standard resolution monitor. This makes it necessary for users to constantly scroll both horizontally and vertically, and leads to considerable user discontent. Some sites have addressed this concern by installing wide monitors on PCs where these screens are accessed. Some Emergency Departments have paid attention to placing PCs in locations that give clinicians most efficient access to information while treating patients Application management There are five Cerner software domains implemented under the emr project, and two additional installations of FirstNet (SSWAHS, CHW) which have been independently implemented. A State Base Build (SBB) domain exists for development and demonstration of the SBB. These FirstNet domains are described below. 29

30 Observations NSCCH AU SEAHS_AU SWAHS_AU North Sydney LHD Central Coast LHD South East Sydney LHD Illawarra LHD Western Sydney LHD Nepean Blue Mountains LHD SSWAHS* Sydney LHD South West Sydney LHD CHW* Children's Hospital at Westmead Have own Cerner implementation - not part of emr project GSGW AU NCAH_AU HNELHD* Far West LHD Western LHD Murrumbidgee LHD Southern LHD Northern NSW LHD North Coast LHD Hunter New England LHD *Not part of emr project - does not run Cerner software. There is considerable variance in each of these implementations, including variance in operating systems, Cerner versions, SBB versions, local configuration, and local implementation of capability outside of the emr scope. Current implementations are summarised in the following table: 30

31 Observations NSCCH_AU HPU'. SEAHS_AU HPU { HPUX 2010, 02 SWAHS_AU HPUX GSGW_AU HPUX H P U X NCAH_AU HPUX SSWAHS + HPUX HPUX Yes X CHW + c i SBB* HPUX HPUX V4 + Not part of emr project - run and manage own software and technology * SBB instance is used for SBB development and demonstration only. This complex and varied environment presents a difficult and expensive application management challenge. Application governance model and processes exist, both locally and centralised The processes to request a change to the State Base Build are well defined and documented. These processes include both local recording and triage processes which then feed into the central triage, recording, assessment and management processes. While these processes are well defined, the local leadership's focus and representation on the governance committees impacts the overall effectiveness of the process. It was reported that ED doctors often find it difficult to attend governance meetings, and this has at times skewed prioritisation and progress toward the interests of those who attend. Multiple versions of infrastructure, Cerner and build across the State adds to management complexity There is a plan and strategy to standardise core FirstNet configurations across the State, however, system changes during the rolling implementation process, and the capacity of sites to localise their systems has driven variances in the implementations at each site. This diversity drives both complexity and operational cost. At some sites new capability cannot be implemented as the base build at that site does not support the required changes. This variance in both the central FirstNet implementation and local integration architecture makes it difficult to get an end-to-end view of any process across the State. This complexity leads to user frustration about the responsiveness of support services, such as the time taken for issue remediation or the effort required to diagnose and respond to a reported issue. Plan in place to gradually migrate regions to a consistent build HSS have developed a plan to migrate the required infrastructure and Cerner versions toward the prerequisites for the delivery of a more standardised State Base Build. This plan includes the upgrading of hardware, operating systems and Cerner versions. Successful execution of this plan should lead to a more uniform set of environments that can accept and support new standardised configurations and capability. 31

32 Observations An ED system requires the integration of a number of systems, including FirstNet, the hospital's Patient Administration System, and various order receiving and laboratory management systems. Ownership of these integrating systems is also shared by HSS and the Local Health Districts. The method by which this integration has been implemented varies, and is dependent on what systems and integration technology exists at each site. We observed a number of integration models at different sites: Centralised integration Local systems integrate directly to the emr SBB integration engine. This has the advantage that there is only one integration infrastructure to maintain, however, it makes the centralised integration engine more complex to manage. Also, there are many more interfaces to manage at the intersection of responsibility between HSS and the Local Health District. The central integration environment must know about all of the implementation details of the local systems. Cerner PAS In sites where the Cerner Patient Administration System (PAS) is deployed, the Cerner software manages all interactions between the clinical modules and the PAS. This considerably simplifies the integration architecture. If additional Cerner modules are deployed, the integration architecture is further simplified. Local integration engine Where a local integration engine is deployed the integration between the centralised and locally managed environments is much simpler and more easily managed. Local Health Districts are more clearly responsible for integration of local systems to the local integration engine. This variety of implementation models increases the overall management complexity of the end-to-end solution and contributes to a general lack of clarity about the management responsibility of the overall architecture. 32

33 Observations 4.4.' Variety of integration models As described above, we found a variety of integration models deployed across the State, as well as variances and combinations of these models. This variety and complexity has added significantly to the overall management effort required to operate and support the end-to-end solution. This variance in models makes it harder to maintain and diagnose issues, since the responsibility for interface management is different for each model and is often not well defined. Varying integration reliability and performance We observed a range of reports on the performance and reliability of interfaces with the emr solution. Some sites reported reliable interfaces and little awareness or concern about failed or lost messages. Elsewhere it was reported that while the integration was reliable the performance was often too slow. And in some sites FirstNet went live without some core interfaces being implemented, and as a result, staff resorted to printing out orders for sending to Pathology for manual re-entry into the pathology system. There were also some reports of interfaces that did not reliably translate and deliver the complete message set. However, in most cases, this was associated with PAS integration and was being addressed and rectified. We did not receive any reports of there being specific adverse clinical outcomes as a result of integration issues. Cerner supports `publish-subscribe' model The FirstNet solution supports a `publish' model for message integration in which messages are `published' to an integration engine for translation and transmission. In this architecture, FirstNet effectively passes all responsibility for the delivery of the message to the integration engine. As a consequence, FirstNet does not necessarily find out if a message fails or contained errors. Equally, there is no feedback to the user if a message has errors or fails. While this is not un-common in systems integration architectures, it places considerably more responsibility on the integration environment and manual processes needed to review and action any interface problems _._ -As and compliance Code and message standards have been generally well applied The use of HL7 for defining messages and SNOMED for the definition of message content (codes) has been generally well applied, and has resulted in a defined and standardised integration architecture. NSW Health has only implemented a restricted set of messages which has aided with standardisation and management of integration, but has in some cases resulted in restrictions on the use of the system. For example, the system only allows `new order' or `cancel order' messages which makes it difficult for users if they wish to make a change to an existing order. A number of clinicians expressed the need to change an order. The maintenance of message and code standards across systems is manually undertaken, which contributes to the effort required to manage overall system integration. Limited visibility of failed messages It was reported that there is little to no user visibility of any messages that contain errors or have failed. In most cases this was not a significant problem for users, who generally found integration to be reliable. In the interface design documentation we observed scant consideration for error processing. In most cases specifications defined error handling functionality with the statement `Use standard error processing'. We expected to find greater detail on what technology, reporting and processes are required to identify and remediate messages that report errors. 33

34 Observations Cerner provide support for outbound messages Cerner is responsible for defining and delivering interfaces into and out of the emr SBB interface engine. The interfaces and message content were designed, documented, built and deployed by Cerner. While Cerner was engaged to develop the message set, the Local Health Districts maintain overall responsibility for reliable message delivery. Local Health Districts provide support for local system integration Any required message translation and integration with local systems was the responsibility of the Local Health Districts, and any relationship they had with their local system suppliers. These interfaces were developed at each site with little coordination across sites. It was observed that most interfacing systems support the required interfacing capability and standards, or have interfaces to local integration engines that enable appropriate integration to the emr SBB integration engine. It was reported that in one example the system used by the Pathology laboratory operator could not match the laboratory results with the corresponding order and encounter within FirstNet (the system could track at a patient identity level only). FirstNet was enhanced to match the result to an encounter; however, the result of this matching is not completely reliable. To compensate, staff check results at the user level, rather than just at the encounter level c:i v < v G. p1 s aki1^ C 111 The primary vendor for the supply and implementation of FirstNet is the Cerner Corporation Pty Limited. The scope of the vendor's responsibility is comprehensively established via three related contracting document types: The original deed of agreement IT-135 between Cerner Corporation Pty Limited "the Contractor" and Health Administration Corporation (i.e. the New South Wales Department of Health), dated 27 September The original deed of agreement provides for hardware acquisition and installation, hardware maintenance, software licensing, IT consultancy, software development and modification, software support, systems integration, data conversion and migration and transition out services. In addition, the agreement specifies a range of mechanisms and measures to govern the relationship with Cerner. GITC Official Orders lodged under the period purchasing agreement IT-135, detailing the scope of services to be provided relating to a specific site implementation. Change Requests covering the provision of software development services or support related to system enhancements or fixes. The vendor's responsibilities appear to be adequately specified via the contracting mechanisms While the requirements for system development and support are formally documented (as described above) the approach taken by NSW Health for managing the relationship with Cerner has generally been less formal. While Cerner provides many of the performance reports as specified in the deed of agreement (IT-35), Cerner has not been formally held to account by HSS in line with the performance governance model specified in the agreements. That said, by most measures it would appear that HSS established an effective working relationship with the vendor. Evidence of this may be found in the measures taken by Cerner to invest in the success of the program, such as providing capability and services beyond that specified in the agreements, and conducting annual reviews by international specialists. 34

35 Observations 'eness A layered governance model operates to identify, assess and escalate to the vendor support and change requests received from SBB sites. User support and the handling of change requests are facilitated via local site representatives. The local user group escalates to the State AAG and ultimately Cerner if necessary. Under the support model, Cerner's responsibility includes third level FirstNet support and the development and implementation of software changes based on direction from both AAG and at times, local areas or sites. Evidence of varying support effectiveness, change management and inadequate code configuration Feedback on overall support effectiveness varied by area and site, and some reports of inadequate system change management and code configuration management were received. For example, users expressed frustration that software changes previously implemented would be lost after subsequent version updates. The implementation approach adopted was that of a rolling model whereby a team would implement the system at a site, including the delivery of initial training, then move on to the next site. The framework for the Change Management and Communications approach was developed centrally by HSS, including the development of tools and templates. Each of the LHDs then localised the approach and materials to suit their specific requirements and took responsibility for delivery. Implementation approach did not allow for follow up reinforcement There is evidence that the time spent by the implementation teams implementing the system and supporting users to learn and become familiar with the system varied by site, and varied according to the size of the site and the number of users. The ED Director of a small rural hospital reported that the implementation team was on site for a total of 5 days. At some of the larger urban hospitals the implementation team was reported as being onsite for weeks prior to and following implementation. There was general agreement that there was insufficient support during the implementations and insufficient follow up support after the implementation team moved to the next site. Some clinicians reported that their hospitals had invested in dedicated support technicians to trouble-shoot and provide support. Consensus on the value and quality of the implementation team Feedback received throughout the review on the value delivered by the implementation teams was consistently high. Clinicians consistently reported that the implementers possessed the requisite knowledge of the system, clinical processes and implementation practices to effectively implement the system. Varying levels of local leadership and investment to support change management and communications Evidence was provided to our review of inconsistent support by senior management at some sites for the implementation process, specifically in ensuring training attendance, implementation of changes to local work practices and investment on local support capabilities. This variation generally correlates with the level of user acceptance of the new system at a site. Transparency of change management lacking When functional changes are made to the system, (either as a result of change requests or version upgrades that have impacted the way clinicians use the system), the process has not always been 35

36 Observations sufficiently transparent. Clinician's `discovery' of unexpected, unannounced or unsupported changes has resulted in varying degrees of frustration.. Change impact of the system on local clinical processes was not adequately addressed There is some evidence that the system was not designed to optimise business processes effectively. Some clinicians reported that the system imposed changes to clinical processes which they regard as suboptimal (for instance, requiring patients to be `clerked' prior to triaging). In contrast to this, some clinicians felt that the system has improved discipline and enforced best practice, albeit with an accompanying increase in the number of steps in some tasks or activities. Limited communication or formal process to support sharing of [earnings Across various sites, as users have become more familiar with the system, local configurations have been made to address many of these usability and process-related problems and enable more effective work practices. Typically, these improved techniques have only benefited the site where they were developed, as there is no formal process or assigned resources to enable improvements that could be shared to be communicated and leveraged by other sites Training effectiveness Consistent with the Change Management and Communications Framework, the framework for training was developed centrally by HSS, including the development of tools and templates. A combination of a `train the trainer' approach was used, with Cerner developing supporting web based training tools. Each of the LHDs then localised the approach and materials to suit their specific requirements. The approach and timeframes for training at each site was agreed between the LHD, implementation team and the clinical management team. This caused a degree of variation amongst sites both within LHDs and across the State. There was no formal user testing or minimum requirement before clinicians adopted the system. Ongoing training and user support continues to be provided by local health services. Varying views on effectiveness of training There were mixed views reported from the clinician community regarding the effectiveness of the training delivered to enable them to adopt and use the system. Some were of the view the content, amount and delivery mechanism were appropriate, while others considered the training to be insufficient, considering the complexity of the system. No formal follow up training program was provided after the implementation to re-enforce learnings and improve the ability of users to leverage the system effectively. Variable ongoing training support based on quality and investment in local training support There are varying levels of investment in training support amongst the LHDs and sites. Some have invested in dedicated local support resources while others relied on services provided by the areas or State. This variation correlated strongly with the perception of quality and availability of training. Attendance at training varied Reports on the levels of attendance at training sessions varied. Reports suggested that nurses and other clinical staff had a higher training attendance rate than doctors. While this was not supported by training attendance data, a number of clinicians commented that it was difficult to get doctors to attend training. This was thought to be due to a number of factors, including the fact that at some sites ED managers made training a mandatory requirement for their nursing staff U Level and clarity of support processes vary by site As previously mentioned, a considerable variance in the user support processes across areas and sites was found. Users reported that it is not always clear who should be contacted to provide support. Some 36

37 Observations sites have recruited dedicated local support resources, some have trained `super-users', while others rely on the regional or HSS centralised support services only. As with ongoing training, support organisations that invested in local 'full-time' support effectively insulated their users from the support processes of the State. This has led to a range of assessments of support from users, from those who are very satisfied with their support processes to those who are considerably frustrated. This variance in support effectiveness has led to considerable variance in user skills, with some doctors being relatively `expert' and able to establish specific lists and shortcuts, while others have only basic user skills. The variation in support also correlates with the degree of local configuration undertaken to overcome usability shortcomings of the system ICT visio - - ^- -ategy FirstNet as a building block in the broader emr program NSW Health is engaged in an emr program that consists of a number of project streams, including FirstNet emr, emr Phase 2, and Medications Management. FirstNet emr (the subject of this review) is completing the implementation of an emr for EDs using components of the Cerner Millennium suite including Cerner FirstNet, Cerner PowerNote, PowerChart, and integration with PAS' s and laboratory systems. A subsequent project stream, emr Phase 2, will build out additional clinical functionality including further enhancements and change requests to Cerner FirstNet, device integration, and further deployment of PowerChart. Another stream is planning to deploy the Medications Management module. A high-level planning roadmap for these and other initiatives is maintained by NSW Health for the purposes of planning business cases and budget submissions over a five year period. Other than this roadmap and the associated business case for funding purposes, we are not aware of any documented vision or strategy that articulates the overall objectives of the emr program, relative priorities or dependencies, or a roadmap to provide more granular planning of the overall program. As a consequence, the program cannot refer to a 'big picture' view that would guide investment decisionmaking, and questions remain about whether the program is pursuing a State-wide emr, multiple emrs, hospital automation, or a combination of these end- states and objectives Business cdz governance As part of the broader emr program, FirstNet must support the State's established governance approach to enable the effective monitoring and utilisation of the resource base. This will require FirstNet to support statutory reporting, and reporting on clinical service delivery and hospital operations. Effective governance requires FirstNet to support adherence to security, user identification, authorisation and audit policies and protocols. Reporting does not enable clinical governance improvements to be measured With a few exceptions, reporting has been flagged as a common area of frustration. Managers reported significant difficulty and frustration in accessing information to assist with managing their ED and other functions, as well as enabling a view of clinical trends. Some clinicians also reported a distrust of the data quality provided in reports, as noted previously. Inadequate user identification and authorisation practices At some sites users reported shared user sessions to avoid delays associated with logging out and then logging back in as a new user. Also, the requirement for clinicians to enter a user identifying PIN number to authorise a transaction has been disabled at some sites. This means that it is not possible to reliably determine the identity of the clinician processing transactions. This issue is exacerbated where clinicians share computers or user sessions. 37

38 Observations Systen- ` System governance processes for managing change requests can be described as follows: Upon identifying a change request the end user logs this with the local user group, sometimes referred to as the local Application Advisory Group (AAG) for review. The local user group reviews each change request to determine whether to put it forward to the FirstNet State AAG. If the submission proceeds, the change request is logged by the State AAG for review and endorsement The FirstNet AAG Triage Group review changes and endorse, delegate to the AAG, or decline each one. All decisions made at Triage are then attached to the following AAG Agenda available for members to review, raise and revisit the decision if necessary If endorsed (as either mandatory or optional) each change request will be placed into scope for the next version of SBB conceptual design. LHDs are able to implement a change request prior to it appearing in the next SBB version. Actual configuration into the LHD is currently the responsibility of the Local Support team. The SBB Team tracks these changes and works with the LHDs to update the Change Control Tracker If a code change is required (i.e. the change request is an enhancement) this is raised with Cerner, which liaises with HSS to ensure the change is made. The system governance process is in place, however, there was evidence this was not effective There is a clearly defined and established system governance process in place (as summarised above), however, a number of clinicians reported frustrations with the effectiveness of this process. Commonly, clinicians reported a perceived slowness and delays in change request implementation. One clinician reported a change request on the FirstNet discharge summary taking 18 months to be implemented. There was evidence that one of the factors contributing to delays of this sort was that the change request process is not adequately robust. In some cases, requirements are not always sufficiently well articulated at the outset leading to rework and delays. Another potential factor is that some change requests are approved by the State AAG but not immediately implemented at the LHD level. Other themes commonly reported include a lack of visibility of progress or status, and the need for many seemingly simple requests to require product enhancements. The imbalance of clinicians and administrators on local AAGs influences change request decisions and priorities Given these and other frustrations with the change request process, some clinicians reported that they had chosen not to participate in the local user groups. Also, some clinicians noted that local user groups were not held at convenient times for clinicians. These factors have contributed to the waning of clinical representation on local user groups. This in turn has influenced change request priorities and decisions. 38

39 Recommendations 5 Recommendations NSW Health has implemented FirstNet as part of the overall emr program to 59 hospitals across New South Wales. While there have been a number of reported issues associated with the deployment, use and operation of the system, many clinicians have reported that it now adds considerable value to the delivery of care and the operation of their Emergency Departments. In the month of March 2011, 86 million transactions were processed; 2.84 million orders were placed and 17,188 users accessed the system on a daily basis. Many of the issues that have been reported and observed relate to system operations, governance, implementation, leadership and training, and not specifically to the capability of the FirstNet system. While the review confirmed that there are a number of issues with the use, implementation, support and configuration of FirstNet, we believe these can be remediated. In deciding the future of the FirstNet solution, it is important to recognise that it comprises an important foundational component of a broader emr architecture, where a patient's medical record may be stored and accessed from within a number of care settings across a continuum of care. Replacing FirstNet would require a new solution to be integrated with the broader Cerner based emr solution set. While this integration is possible, it would add an additional level of complexity into what is already a very complex environment. A range of issues and frustrations were identified across the FirstNet sites examined, which are notably impacting user satisfaction and the effective use of the system. These issues highlight the urgent need for a formal program of site specific remediation activities. These needs might include remedial training, configuration and in some cases, upgrades to the currently implemented system. A well defined plan and program of work needs to be urgently put in place to bring all sites to a base level of acceptable infrastructure, functionality, useability and user training. From this context, we strongly recommend that the following activities be performed: A detailed independent review should be undertaken of each site to identify what issues exist and what remediation activities are required to elevate the system and its use to a defined minimum standard. This minimum standard should be based on the capabilities delivered in the latest version of the SBB. This review should be executed as a matter of urgency. We further recommend that this remediation program be delivered by resources other than those already responsible for the current implementations or the planned emr phase 2 to avoid resource conflicts with these activities. [an 39

40 Recommendations plan A formal project should be established and funded to implement the recommendations set out in the remediation plan. There is currently considerable variance across each of the implementations of FirstNet in the LHDs. While NSW Health has adopted the concept of a State Base Build (SBB) - many of the implementations are deployed on different operating system versions; different Cerner versions; different SBB versions; different local configurations; and in some cases, local additions that are outside of the scope of the emr program. This is driving inefficiency and complexity in the use, operation, support and management of these systems. To address this issue, we propose the following recommendations: Define and expand tl^ _ ` - ^ cope To the extent that is possible, NSW Health should expand the scope of the SBB to include many of the currently variable components of the local implementations, and then standardise where appropriate the implementation of the SBB across the LHDs. This will drive greater efficiencies of operation and enable the sharing of many of the local configurations that are not currently available to all Develop a plan for expansion A plan must be developed for the definition and implementation of all the components required for the new SBB. This plan must take into account any remediation work that is required to address as many as possible of the issues associated with the current implementations while taking into account any planned implementation and enhancement activities, including the implementation of emr Phase 2 and Medications Management. Analysis should also be undertaken to determine how those sites that are currently not part of the State- Base Build program i.e. non-sbb sites, would be managed going forward C- to implement emr 2 and medications ionality There are currently plans and budget submissions for the implementation of the emr phase 2 and medications management capability. This includes the addition of new clinical support functions as well as the integration of `real time' monitoring devices into the Cerner system. It is our view that NSW Health should progress with these plans to implement this new capability in alignment with the approach recommended for the new and expanded SBB E Considerable dissent was expressed during the interviews regarding the reporting capability that is provided by the current implementation of FirstNet. ED Directors and hospital managers need access to the rich information that exists within FirstNet to better manage the day-to-day activities under their responsibility and to understand how their clinical and administrative strategies are performing. NSW Health must establish a reporting capability that readily enables the development of new reports and enables managers to personally interrogate the available information. Our initial investigations suggest that the resolution of this issue will include a review of the use of data input validation rules, the reporting tools available and training of users in the use of these tools. 40

41 Recommendations t SBA lice In developing a more standard approach to the definition, implementation and management of the SBB, a clear and effective communication plan must be developed and put in place. It will become increasingly important that all SBB stakeholders are well informed of strategies, new capability, and current activities to improve the FirstNet system. ovation ` C ins to A number of situations were observed where useability problems experienced by one or more users had been solved by local configuration or modified practices at another site. An exercise should be undertaken to establish where such changes have taken place and how these may be incorporated into the next release of the SBB and shared across the sites. A longer term strategy and process should be put place to enable all stakeholders to understand and share any local innovations and configurations. NSW Health has made substantial steps toward the implementation of an emr for all patients presenting at hospitals across the State. The emr will provide a basis for participating in the broader national Personally Controlled Electronic Health Record (PCEHR) program, while providing the longitudinal record of patient care essential for addressing the ever increasing demands on the health system. The FirstNet solution and the broader emr program provides key tools for NSW Health and the Local Health Districts to enable the delivery of improved care across the full continuum of care, which often starts at the Emergency Department. These tools provide clinicians with improved access to information to support the treating of patients, and to understand how clinical processes can be changed to improve patient treatment outcomes and the utilisation of resources. With this is mind, we would recommend the following: a Chief Medical Information Officer (C_MIO) role As IT systems are increasingly used to support the delivery and improvement of health care, it is important that senior clinical input drive the vision and requirements for the use and future development of clinical systems. We recommend the creation of a Chief Medical Information Officer (CMIO) in NSW Health to provide this direction. The Chief Medical Information Officer (CMIO) role should be established to provide the additional leadership required to ensure that the future implementations and operation of FirstNet and other clinical systems are successful. This is an executive role which is responsible for the development of a vision for the capabilities necessary to enable the care delivery strategies of the Department Oil Establish a vision for the clinical processes and necessary systems capability required to enable the care delivery strategies of the Department. This will become increasingly important as patients' medical records become more electronic, and start to be shared outside of the department, including with the Personally Controlled Electronic Health Record (PCEHR). This vision should provide the basis from which all systems implementation strategies will be defined and developed into implementation roadmaps. This vision will be fundamental to the successful delivery of future clinical based systems. 41

42 Recommendations The success of systems introduced to support clinical functions requires appropriate funding for the system's initial implementation and on-going operation. The review identified that the funding model established for the FirstNet program have been very project driven. While this approach has successfully supported the initial implementation of the system, it has not expressly provided for the iterative retrospective application of enhancements resulting from lessons learned and ongoing clinical process improvement. More specifically, the on-going system development and training support post implementation appears not to be expressly provided for under the current funding model. This negatively impacts user satisfaction and adoption and the harvesting of benefits expected from the system Review program f models It is therefore important that funding for the future phases of the emr program includes both implementation and on-going operating costs. There are multiple parties involved in the implementation and delivery of FirstNet. HSS has responsibility for delivery of the core application, integration and infrastructure; the LHDs have responsibility for local systems, local infrastructure and systems integration; Cerner provides application implementation and support services; and the Department provides governance, funding and leadership support. Evidence suggests that the specific responsibilities of each organisation involved in the implementation of FirstNet and the broader emr program are not well defined and understood by all stakeholders. This uncertainty has driven inefficiency and stakeholder discontent Establish and t_*-._ nt program governance and organisational structures An assessment of the governance and organisational structures required to effectively deliver a computer enabled clinical care system must be undertaken. Any changes to organisation structure, delivery and support model, and governance processes must be clearly defined and implemented. There was considerable discontent expressed regarding the visibility and effectiveness of the system operation and user support processes. Users are often uncertain as to who is responsible for addressing support issues and what the proper process is for raising requests for change. The level of local support available also varied greatly from site to site; in some cases the local organisations had clearly invested significantly to establish local expertise and support capability, while others relied mainly on the centrally delivered support services. 42

43 Recommendations Visibility and feedback on the status of requests for support or change is also an issue. It was reported that in many cases, requests were made and little or no feedback was received, and some cases no action was ever taken EstabIis' and transparent operation and uses t processes It is important that the Department establish the system operation and user support process required to ensure that users of FirstNet achieve the maximum benefit from the use of the system. In addition, users should have access to appropriate ongoing training to ensure they have the skills to effectively use the system. These redefined processes must ensure that the users of the system have clarity, transparency and certainty about any requests they may make. As the hospitals across the state move more towards a paperless care delivery model, the reliance on the availability of the clinical systems and the patient information in those systems becomes critical to the clinicians ability to deliver effective care. It is essential that highly available and high performance infrastructure be provided to support the delivery of this increasingly critical IT service. Evidence suggests that the existing infrastructure does not universally meet this minimum standard. We therefore recommend that the following activities be undertaken Define infrastructure delivery approach A review should be undertaken of the infrastructure delivery approach, and an assessment made as to the suitability and capability of each contributing party. This review should also include an assessment of alternate sourcing approaches. Many international users of Cerner do not run their own data centres, and outsource the hosting and management of the core infrastructure required to run the application to specialist third parties P..' A'_n-.. t_ )rt model Evidence suggests that the support processes and responsibilities for the delivery of infrastructure support services are not clearly defined and understood by many users. The infrastructure support model and the responsibilities of all parties involved in the delivering support should be clearly defined and communicated Define " -` -f u 1 There is a broad variety of technologies deployed to enable the delivery of FirstNet. The size and location of PCs varies widely; there are multiple network and application integration designs; and a range of server environments and operating systems. A Standard Operating Environment (SOE) should be defined and included in all future upgrade and implementations plans. 43

44 Appendix A Appendix A - Interview List We interviewed and/or conducted meetings with the following people in the conduct of this engagement: Name Position Interview date(s) Tony Azzam emr SBB Program Manager 24-May & 14-Jun Ronan Herlihy Benefits Realisation Mgr - HSS 24-May John Frisken CTO - Unitech Solutions Group 25-May Dr Michael Barnet Director - Meridian Health Informatics 25-May Prof Jon Patrick Professor - University of Sydney 26-May Dr Tim Smyth Deputy Director-General, NSW Health 26-May Howard Dawson Education co-ordinator - HSS 30-May Cameron Burt General Manager - Cerner 31-May & 27-Jun Prof Joanna Westbrook Joanne Callan Director, Centre for Health Systems and Safety Research - UNSW Senior Research Fellow - Centre for Health Systems and Safety Research - UNSW 1-Jun 1 -Jun Greg Wells Chief Information Officer - HSS 2-Jun & 28-Jun Tim Hume Director, Strategy and Architecture - HSS 2-Jun Dr Richard Paolini Anthony Futia Head of Emergency Department, Concord Repatriation Hospital Data Integrity liaison Officer and ED Data Collection Co-ordinator Barbara Howell Clinical Manager - Emergency Department - Wollongong Dr Tom Corrigan Staff Specialist- Emergency Department - Wollongong Kelly Peterson emr Project Director - Northern Sydney Local Health District 2-Jun 3-Jun 7-Jun 7-Jun 8-Jun 44

45 Appendix A Dr David Rivett GP Proceduralist - Batemans Bay 8-Jun Dr Alan Forrester Karen Braid Dr Rob Davies Dr Rod Bishop Dr Trevor Chan Leanne Ovington Dr Linda Dann Chair FirstNet AAG/ Network Director Emergency Services Clinical Nurse Consultant Clinical Projects & Informatics - Nepean Blue Mountains Hospital Staff Specialist, Emergency Dept - Tweed Hospital Co-chair MTEC / Staff Specialist, Dep't of Emergency, Nepean Blue Mountains Hospital Staff Specialist, Emergency Dept, St George Hospital ED/HDU Nurse Unit Manager, Moruya / Batemans Bay Hospitals Director Emergency Department, Bankstown Hospital 8-Jun 9-Jun 9-Jun 9-Jun 9 -Jun 10-Jun 10 Jun Director of Medical Service & Director, Medical Dr Colin MacArthur Assessment Unit & Staff Geriatrician, Liverpool 10-Jun & 23-Jun Hospital Dr Randall Greenberg Director emergency Department, Dubbo Hospital 14-Jun Dr Matthew Vokasovic Emergency Department Director, Westmead Hospital 15 Jun Dr Sally McCarthy Medical Director, Emergency Care Institute 15-Jun Dr Mathew O'Meara Dr Michael Hession Rosemary Beenie Dr Adam Chan Dr Roger Trail Director, Emergency Department, Sydney Children's Hospital Network, Randwick Staff Specialist, Emergency Dept, Blacktown Hospital Nurse Manager Clinical Informatics, Liverpool Hospital Director, Emergency Department, St George Hospital Anaesthetist, Department of Anaesthesia, Royal Prince Alfred Hospital, Camperdown 16-Jun 16-Jun 17-Jun 23-Jun 24-Jun 45

46 Appendix B Appendix B - Acronyms HSS SBB emr EMR AAG LHD SSW CHW SSWAH NSCCH SWANS GSGW NCAH HNELHD SEAH PAS ECG RIS PACS JCAPS SNOMED CMIO ED AHS UI Health Support Services State Base Build NSW Health program to implement Cerner Millennium suite of products to enable support for clinical functions Electronic Medical Record Application Advisory Group Local Health District Sydney South West Children's Hospital Westmead Sydney South West Area Health North Sydney Central Cost Health South West Area Health Service Greater Southern Greater Western Northern Coast Area Health Hunter New England Local Health District South East Area Health Patient Administration System Electro Cardio Gram Radiography Information System Picture Archiving & Communication System Java Composite Application Platform Suite Systemised Nomenclature of Medicine Chief Medical Information Officer Emergency Department Area Health Service User Interface 46

47 Appendix B PCEHR SOE ICT GITC Personally Controlled Electronic Health Record Standard Operating Environment Information and Communication Technology Government Information Technology Condition 47

48 Appendix C Appendix C - Source Documents 1 Cerner NSW Health Contract HSS 2 State Baseline Build - Emergency Department Design Document HSS 3 EMR Architecture Strategy HSS 4 Contractor Specifications HSS 5 A Critical Essay on the Deployment of an ED Clinical Information System - Systemic Failure or Bad Luck? Internet 6 Electronic Medical Record - State Baseline Build - Approach to Content Development HSS 7 Final Report of the Special Commission of Inquiry - Acute Care Services in NSW Public Hospitals Internet 8 Electronic Medical Record - Post Implementation Benefits Tracking Summary Report HSS December "Lights On" Information - Northern NSW & Mid North Coast HSS 10 Orders for all sites 1. GSAHS Cerner_official order_cerner GWAHS Cerner_official order_cerner 3. North Coast Area Health Service 4. South Eastern Sydney Illawarra Area Health Service 5. Sydney South West Area Health Service 6. Sydney West Area Health Service 7. The Children's Hospital at Westmead HSS NSW HEALTH VALUE REVIEW HSS 12 EMR PACSRIS ROLLOUT (as at 4 May 2011) HSS 13 Cerner Millennium Support Model HSS 14 Client Profile Metrics Explanations HSS Q4 SolutionWorks Millennium Applications Client Dashboard HSS 16 Tender Evaluation Document HSS EMR Master File All Finalised RFT documents 1. Appendix E_H_I_J_K_EMR_IT Appendix A - Tenders Response Forms 3. Appendix B Master HL7 Ref Guide 4. Appendix C_IPS IT 189_EMR_IT Appendix D GITC RFT it HSS 48

49 Appendix C 6. Appendix E_Area Configurations_EMR_IT Appendix F_Demonstration Scripts_EMR_IT Appendix G HL Compliance Resolutions_EMR_IT Appendix H - Software Acceptance Testing_EMR_IT Appendix I _I PS Methodology_EMR_IT_ Appendix J- Definition of Terms_EMR IT Appendix K_Non negotiable_emr_it EMR RFT 190 Final Version 18 Certificate of Registration - Quality Management System - ISO 13485: 2003 Cerner 19 Certificate of Registration - Quality Management System - ISO 9001: 2008 Cerner 20 How does information technology impact on quality of care? Johanna Westbrook What impact do emergency department information systems have on nurses' access to information? A qualitative analysis of nurses' use and perceptions of a fully integrated clinical information system Use of information and communication technologies to support effective work practice innovation in the health sector: a multi-site study Johanna Westbrook Johanna Westbrook 23 Cerner consulting and managed services HSS 24 EMR2 Final Business Case HSS 25 EMM Business Case Final HSS 26 Cross Client Reporting: User Experience All Positions Cerner 27 Cross Client Reporting: Client Profile Report Cerner 28 Lights on Network Usability Dashboard User Guide (PowerOrders) Cerner 29 Australian IP Overview Cerner 30 Meetings Overview Cerner Q4 SolutionWorks Millennium Applications Client Dashboards Cerner 32 Milestone Management Updated Cerner 33 NSW_AU Milestone Management Cerner 34 EMR Domain Status HSS 35 Implementation Status of Strategic Products or Applications at each local Health Network HSS Q4 SolutionWorks Millennium Applications Client Dashboard Client # / CernerWorks Client / Greater Southern Area Health Service (Queanbeyan, NSW) 2010Q4 SolutionWorks Millennium Applications Client Dashboard Client # / CernerWorks Client / Greater Southern Greater Western (Chatswood, NSW) 2010Q4 SolutionWorks Millennium Applications Client Dashboard Client # / Not CernerWorks Client / Health Support Services (NSW) (North Sydney, NSW) Cerner Cerner Cerner 39 "Lights On" Data - NSW Health - User Experience -April Cerner 40 Lights on Data June 10 Cerner 41 Lights on Useability April Cerner 49

50 Appendix C 42 NSW_AU Container Membership Report Cerner Q4 SolutionWorks Millennium Applications Client Dashboard Client # / CernerWorks Client / North Coast Area Health Service (Lismore, NSW) 2010Q4 SolutionWorks Millennium Applications Client Dashboard Client # / CernerWorks Client / Northern Sydney Central Coast (Gosford, NSW) 2010Q4 SolutionWorks Millennium Applications Client Dashboard Greater Southern Area Health Service (Queanbeyan, NSW) Greater Southern Greater Western (Chatswood, NSW) Health Support Services (NSW) (North Sydney, NSW) North Coast Area Health Service (Lismore, NSW) Northern Sydney Central Coast (Gosford, NSW) Health Administration Corporation NSW (North Sydney, NSW) 2010Q4 SolutionWorks Millennium Applications Client Dashboard Client # 855 / CernerWorks Client / Health Administration Corporation NSW (North Sydney, NSW) Cerner Cerner erner Cerner 47 NSW emr Learning Strategy HSS 48 emr Lessons Learned HSS 49 Final Training Numbers Discipline NCAHS HSS 50 elearning.htech.health.nsw.gov.au ( ) HSS 51 elearning.htech.health.nsw.gov.au ( ) HSS 52 elearning.htech.health.nsw.gov.au ( ) HSS 53 elearning.htech.health.nsw.gov.au ( ) HSS 54 Statistical Data on Class Attendance - Sector HSS 55 SESI emr Training Evaluation HSS 56 Electronic Medical Record Training Evaluation HSS 57 South Eastern Sydney Illawarra health Learning Plan HSS 58 Sydney West Area Health Service Learning Plan HSS 59 Greater Southern Health Learning Plan HSS 60 Greater Western Area Health Service emr Learning Plan HSS 61 North Coast Area Health Service Learning Plan HSS 62 Learning Plan Development Session (LPDS) Agenda HSS 63 LPDS Gap Analysis HSS 64 Risk Assessment Tool HSS 65 The Learning Plan Development Session HSS Emergency Dept Discharge Summary SCh Triage 4 Performance Children's Hospital Randwick Children's Hospital Randwick 50

51 Appendix C 68 What attributes did Cerner have in Oct 2002 that gave it Value? HSS 69 Greater Southern and Greater Western Area Health Service emr Business Case HSS 70 GWAHS and GSAHS Project Charter emr Implementation HSS 73 emr Post Implementation Action Plan_Goulburn HSS 74 emr Post Implementation Action Plan_Queanbeyan HSS 75 Greater Southern Area Health Service (GSAHS) emr Post Conversion Assessment HSS 76 GSGW - Bathurst Base Hospital Deliverables HSS 77 Greater Western Area Health Service (GWAHS) emr Post Conversion Assessment HSS 78 Cover Letter - PIR for Goulburn Base Hospital HSS 79 Attendance list for User Acceptance Testing HSS 80 High level timeline GSGW Baseline HSS 81 Attendees - NCAHS FirstNet Advisory Group HSS 82 North Coast Area Health Service emr Business Case (Part A) HSS 83 NCAH PCA FirstNet Deliverables HSS 84 North Coast Area Health Service NCAHS emr Post Conversion Assessment HSS 85 NCAHS Event Evaluation HSS 86 Post conversions review trip report_ncahs HSS 87 North Coast Area Health Service Project Charter HSS 88 FirstNet UAT Attendees HSS 89 Northern Sydney Central Coast Area Health Service Project Charter HSS 90 Steering Committee Meeting Papers HSS 91 South Eastern Sydney Illawarra Area Health Service emr Business Case HSS 92 Feedback - SESI HSS 93 SESIAHS - Project Charter Comments HSS 94 SESIAHS End System Validation Assurance Report Sign Off HSS 95 Post Conversion Report - Implementation of the Electronic Medical Record at St George Hospital HSS 96 South Easstern Sydney Illawarra emr Readiness Assessment Sign Off HSS 97 Post Conversion Report - The Sutherland Hospital HSS 98 The Wollongong Hospital - Electronic Medical Record Post Conversion Asssessment Report HSS 99 SESIAHS Event Attendance Sheet HSS 100 SESAHS - AU Event Attendance Sheet HSS 101 Terms of Reference - SESIAHS FirstNet Clinical Advisory Group HSS 102 SWAHS emr Stack I Project Summary HSS 103 Approved Plan A HSS 51

52 Appendix C 103 Attendance FirstNet SME Group HSS 104 Chang Register FN Mad Nurse Clerical HSS 105 Cerner Event Activity Report HSS 106 Area Health Service FirstNet Application Advisory Group HSS 107 Sydney West Area Health Service Project Charter HSS 108 SWAHS Stack 2 Rollout Plan HSS 109 Test Level Completion Report HSS 110 SWAHS Event Activity Report HSS 111 Health Technology DietOrders_Integration_Components HSS 112 EMR Diet Order Interface HSS 113 RIL Interface High Level Architecture HSS 114 Health Support Services EMR Integration Plan HSS 115 GW/GW ipm - EMR ADT Interface HSS 116 EMR QR Interface HSS 117 Health Support Services Integration Design - S.E.R Orders and Results HSS 118 Health Technology Interface Design -ADT Messages HSS 119 Interface Design -ORM Messages HSS 120 GW AHS - GW AHS Radiology Interface Plan HSS 121 EMR egate NC Network Diagram HSS 122 EMR NC AHS Order Interfaces HSS 123 Health Technology Integration Design - NC AHS Orders HSS 124 Health Support Services - Integration Design CRIS Orders and Results HSS 125 NSCC Path Net Orders Interface Design HSS 126 NSCC Diet Order Interface Architecture Diagram HSS 127 Interface Project Plan - NSCC EMR HSS 128 NSCC Millennium (EMR) Ward/ED/PAC HSS 129 NSCC Millennium (EMR) Ward / ED / PAC HSS 130 Millennium Foreign System Interface: GSAHS and GWAHS HSS 131 Vision Software Technologies HL7 Interface Specifications HSS 132 Vision Software Technologies Royal North Shore (RNS) HL7 Interface Gap Analysis HSS 133 EMR QR Interface HSS 134 EMR SBB Integration Engine HSS 135 The PRB Problem Detail Segment HSS 136 Millennium Foreign System Interface: SESIAHS Version 1.1 HSS 137 Millennium Foreign System Interface: SESIAHS Version 1.4 HSS 52

53 Appendix C 138 Millennium Foreign System Interface, Foreign System i.pm Version 1.1 HSS 139 Millennium Foreign System Interface, Foreign System Omnilab Version 1.1 HSS 140 Millennium Foreign System Interface, Foreign System PMI/ADT Upload Version 1.0 HSS 141 Millennium Foreign System Interface, Foreign System e*index and i.pm Version 1.2 HSS 142 FSI HL7 Universal Interface Specifications Unit20BBlood Bank Permanent Patient Data and Blood Bank Product and Transfusion History Upload to PathNet Millennium HSS 143 Foreign System Interfaces HL7 Universal Interface Specifications HSS 144 Foreign System Interfaces HL7 Universal Interface Specifications Unit 8i; ADT HSS 145 Foreign System Interfaces HL7 Universal Interface Specifications Unit 90: Order Message Processing Outbound HSS 146 Health Technology SWAHS EMR Interface Plan HSS 147 List of External Systems Interested in the HL7 message testing HSS 148 SWAHS Millennium Prod Interface Audit HSS 149 Health Technology TSS Design - AD Design for Application Delivery by Citrix HSS 150 EMR Domains and Servers - July 23 HSS 151 EMR High Level Infrastructure Design V1.1 HSS 152 emr Summary Transaction Statistics HSS 153 North Coast Area Health Services - HNAM High Availability Operational Guide HSS 154 Sydney West Area Health Services - HNAM High Availability Operational Guide HSS 155 Northern Sydney Central Coast - HNAM High Availability Operational Guide HSS 156 SESIAHS EMR Performance Report Post St George Go Live HSS 157 AHS emr Quality Plan HSS 158 FirstNet AAG Minutes ( ) HSS 159 What is SBB? HSS 160 SBB FirstNet Change Request Tracker HSS 161 SCh Triage 4 Performance HSS 53

54 Appendix D Appendix D Chronology of Events March 2000 Report of the NSW Health Council issued 2002 Point of Care Clinical Business Case issued and approved, partially funded September 2002 May 2005 Dec 2005 Original deed of agreement with Cerner signed emr tender released to explore options for an alternative emr solution Evaluation report complete May 2007 Former Sydney South West Area Health Service: First site goes live - Non State Base Build - Liverpool 14 May 2008 Former Sydney South West Area Health Service: First site goes live - Non State Base Build - Bankstown 30 September 2008 Former South East Sydney & Illawarra Area Health Service: First site goes live for the State Base Build emr implementation (including FirstNet) - St George 24 November 2008 Former North Coast Area Health Service : First site goes live for the emr implementation (including FirstNet) - Lismore 18 March 2009 Former Sydney West Area Health Service : First site goes live for the emr implementation (including FirstNet) - Westmead 24 March 2009 Former Sydney West Area Health Service: Last site goes live for the emr implementation (including FirstNet) - Nepean 26 October 2009 Former South East Sydney & Illawarra Area Health Service: Last site goes live for the emr implementation (including FirstNet) 26 October 2009 Former North Coast Area Health Service: Last site goes live for the emr implementation (including FirstNet) - Port Macquarie 12 May 2010 Former Greater Southern Area Health Service: First site goes live for the emr implementation (including FirstNet) - Goulburn 15 June 2010 Former Greater Western Area Health Service: First site goes live for the emr implementation (including FirstNet) - Bathurst 6 July 2010 Former Greater Western Area Health Service: Last site goes live for the emr implementation (including FirstNet) - Dubbo 28 July 2010 Former Greater Southern Area Health Service: Last site goes live for the emr implementation (including FirstNet) - Wagga Wagga 54

55 Appendix D 12 April 2011 Former Northern Sydney Central Coast Area Health Service: First site goes live for the emr implementation (including FirstNet) - Gosford About beloitte Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and independent entity. Please see for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms. Confidential - this document and the information contained in it are confidential and should not be used or disclosed in any way without our prior consent Deloitte Touche Tohmatsu. All rights reserved. 55