Showcase Hospitals Local Technology Review Report number 3. Quality Compass

Showcase Hospitals Local Technology Review Report number 3 Quality Compass

The Healthcare Associated Infections (HCAI) Technology Innovation Programme The basic ways of preventing and reducing healthcare associated infections (HCAIs) are largely unchanging. The principal strategies for combating HCAIs are those associated with hand hygiene/aseptic techniques, prudent antibiotic prescribing and good clinical practice. However, new technologies and equipment can support these strategies by helping get things done differently, more swiftly or more reliably. The Department of Health is funding the HCAI Technology Innovation Programme 1. The Programme aims to Speed up the development and adoption of technologies to further help combat HCAIs Identify which new technologies provide the best value and will have the most impact The Showcase Hospitals Programme As part of the HCAI Technology Innovation Programme, Showcase Hospitals are undertaking local technology reviews of infection related products or technologies in which they have a specific interest. These are service evaluations, as defined by the National Patient Safety Agency s National Research Ethics Service, and do not therefore require Research Ethics Committee review. 2 This service evaluation was undertaken by The Royal Wolverhampton Hospitals NHS Trust. 1 For further information on the Programme see http://www.clean-safecare.nhs.uk/index.php?pid=28 2 See leaflet on defining research at http://www.nres.npsa.nhs.uk/news-andpublications/publications/nres-research-leaflets/

Acknowledgements We would like to acknowledge the support of the NHS Technology Adoption Centre in the compilation of this report.

Showcase Hospitals Local Technology Review report number 3 Quality Compass Contents Executive summary 2 Introduction 3 The problem The need to co-ordinate information about healthcare-associated infections (HCAIs) 3 The product Quality Compass 3 The knowledge base What was known before this evaluation 4 The evaluation The aim and objectives of the evaluation 5 Timing of the project 5 Issues which arose prior to installation of the system 5 Installation of the system 7 Post installation 7 How acceptable was the product to the Infection Prevention Team? 13 Advice for trusts considering introducing Quality Compass Important points to consider 14 1

Executive summary As part of the Department of Health s Healthcare Associated Infections (HCAI) Technology Innovation Programme, Showcase Hospitals are undertaking local technology reviews of infection related products or technologies in which they have a specific interest with the objective of helping Directors of Infection Prevention and Control and other stakeholders to decide whether they should consider any of these products or technologies as part of their trust s strategy to reduce healthcare associated infections. The extraction, collation, and analysis of data from a number of different hospital systems are inefficient. There would be benefit in a system which brought together relevant data from the different systems and included a set of tools to make data analysis easy. The Royal Wolverhampton Hospitals NHS Trust decided to review Quality Compass which is a Web-based technology platform that is intended to allow for immediate diagnosis of individual episodes of care thereby creating opportunities to enhance patient safety, improve outcomes, and reduce costs. The project ran for a year from October 2008, with the system being ready and available for evaluation from June to October 2009. Over the course of the project, it did not prove possible to resolve a number of issues relating to the integration of Quality Compass with the Trust s IT systems. This meant that infection prevention staff had to cross check all information with their existing systems. Staff could, however, see the potential of the system if it proved possible to resolve these issues, and found it easy to use. The issues which arose with the local IT systems during this evaluation suggest that Trusts should be cautious in considering the use of Quality Compass. They should ensure as far as possible that their own systems will be able to interact effectively with Quality Compass before investing effort in seeking to adopt it. 2

Introduction This report sets out the findings from an evaluation in The Royal Wolverhampton Hospitals NHS Trust (the Trust), one of eight Showcase Hospitals, of the in-use features and adoption characteristics of Quality Compass. The objective of this document is to help Directors of Infection Prevention and Control and other stakeholders, including trust IT departments, to decide whether they should consider Quality Compass as part of their trust s strategy to reduce healthcare associated infections. The problem The need to co-ordinate information about healthcare-associated infections (HCAIs) Hospital infection prevention teams (IPTs) currently have to manually extract, collate, and analyse data from a number of different hospital systems. This is inefficient. Efficiency would be improved if IPTs could make use of a single system which brought together relevant data from the different systems and included a set of tools to make data analysis easy. The product Quality Compass Quality Compass uses QC PathFinder, a Web-based technology platform developed by Vecna Technologies Inc. Quality Compass is provided by The Advisory Board Company (ABC). It is intended to allow for immediate diagnosis of individual episodes of care thereby creating opportunities to enhance patient safety, improve outcomes and reduce costs 3. In the United States Quality Compass comprises a national network of hospitals and health systems which aims to create a shared platform and common set of metrics for robust infection related performance benchmarking and data mining. It is not yet established in the UK. The system incorporates various tools aimed at providing the user with effective means to prevent and control infection, including: Dashboards and Score Cards track and trend performance vs. targets for key indicators via role specific dashboards Performance Alerts Rapid identification of emerging problems to trigger timely intervention Online Reports make detailed performance information immediately available across the entire organisation 3 Information about the programme is not currently available on the ABC website www.advisory.com as ABC is selective about the organisations to which the programme is offered. Information can be obtained by emailing qualitycompass@advisory.com. Information is also available on the Vecna Technologies website www.vecnamedical.com. 3

Drill down analysis Drill down from high-level metrics to the individual patient account level to isolate the root causes of observed problems Ad-Hoc Querying Filter large transactional data sets to identify the specific accounts that meet user-specified criteria Comparative Benchmarking Compare performance across multiple dimensions, such as locations, physicians, procedures Coupled with these tools are: Dedicated Advisor Service ABC consultant assigned to each facility to provide training and guidance Sharing best practices and facilitating teleconferences and meetings. Through best practice sharing, partners have the opportunity to learn how others are leveraging Quality Compass to reduce infections and optimise antibiotic use Before the project began, the Trust and ABC signed a letter of agreement that, among other things, stated that the Quality Compass web-based business intelligence tool would include: Customised view and analysis of the Trust's own infection data (clinical and financial) Ability for multiple users to easily access and track key performance metrics on demand Detailed, drill-down reporting and analytics Quality Compass would also include access to a full complement of resources and services aimed at fostering networking and sharing knowledge of how member hospitals are leveraging Quality Compass to drive better clinical, operational, and financial results, including: Case studies profiling hospital infection reduction successes Research briefs on infection prevention and treatment best practices User group conference calls to share success stories on use of the tool to drive clinical and financial impact. The knowledge base What was known before this evaluation No previous publications have been identified. 4

The evaluation The aim and objectives of the evaluation The aim of the project was to introduce, use, and evaluate Quality Compass over a 12-month period. The objectives of the project were to establish the extent to which the Enhanced Surveillance Capabilities of Quality Compass could: Be used to enhance and improve the Trust s existing infection prevention related surveillance methods and subsequent reporting Be compared with existing surveillance methods Align with the Infection Prevention Team s existing daily work commitments/routines. The intention was to gain insight into the following areas: Quality of training received Whether it was possible to create useful reports using Quality Compass Whether the reports generated in Quality Compass better informed dialogue with executive and clinical staff Its ease of use and also navigation of the site to create useful reports Whether the educational opportunities offered by the ABC were utilised Whether Quality Compass provided data that was not available prior to its use Ease of access to and quality of support offered by the ABC Time taken to collect and input data Quality of customer service received IT compatibility IT stability and data security. It should be noted that, given the length of the project, there was no intention to assess whether the use of Quality Compass enhanced patient safety, improved outcomes and reduced costs. Timing of the project The project ran for a year from October 2008, with the system being ready and available for evaluation from June to October 2009. Issues which arose prior to installation of the system Prior to installation of Quality Compass, discussions were held with ABC about what the system could achieve and what would need to be in place in order for the system to be used effectively. The ABC team were knowledgeable and helpful. In the course of these discussions the Infection Prevention Team (IPT) were keen to investigate whether certain databases already in use across the Trust 5

could be accessed using Quality Compass as a one stop shop to gather information. For example, if an MRSA bacteraemia occurred and the user was alerted to the result through Quality Compass, that user could then ascertain: What the appropriate Saving Lives High Impact Intervention scores were for that particular clinical area Environmental and Hand Hygiene audits results Health Act balanced score card status Isolation room usage data Both the local IT team and ABC agreed that this should be possible. A dedicated advisor was available from ABC who worked closely with the Trust during the set up and then on to discussing the system with the IPT. Although the advisor was based in the US, the IPT were offered teleconferences at mutually convenient dates/times and ABC also visited the Trust in person when in the UK. Table 1 sets out issues which arose in the course of these discussions, what action was taken to resolve them, and the outcome of that action. Issue Action Outcome As there was no opportunity for the duration of the project to be extended, ABC expedited the build process to meet the project s short timeline. 1.1 ABC indicated that they usually allowed approximately 18 months for the installation process, whereas this project aimed to introduce, use and evaluate Quality Compass over a period of 12 months. The reduced length of time for installation may have contributed to some of the problems that arose in use. In particular, no user acceptance testing took place 1.2 Although Quality Compass is intended to draw upon information already present in the hospital such as the Patient Administration System (PAS) and the hospital laboratory system, it also involves a significant amount of manual data entry to enable the surveillance to be effective. Note that Quality Compass is based on a clinical judgement An audit and surveillance nurse was employed and the Trust IPT decided to focus on a limited range of areas in order to ensure that the areas to be covered were resourced adequately. The topics for surveillance were: Ventilator Associated Pneumonia (VAP)-Critical care Urinary Tract 6 It became apparent that it would be difficult to monitor the VAP rate as at that time there was discussion ongoing as to what definition to follow. The audit and surveillance nurse predominantly used the system to track catheter associated UTI episodes on four general wards and worked with the DIPC in investigating hospital acquired bacteraemias.

model requiring a user to confirm automated Infections-Critical care infection designations. Urinary Tract Infectionsgeneral wards Hospital acquired bacteraemias. Table 1: Issues that arose prior to installation of the system In the light of these discussions, the Trust team felt that the installation should have been reasonably uneventful. Installation of the System During the installation phase, the local IT team continued to hold face to face and teleconference meetings with ABC. The installation generally went well. In terms of technical specifications, ABC provided the IT team at the Trust with the relevant Health Level Seven International (HL7) 4 documentation. This documentation was of a high standard and met requirements. The Quality Compass system was hosted by ABC. The local IT team at the Trust were responsible for setting up a secure File Transfer Protocol (FTP) link to link to the system. Full documentation was provided by ABC at the outset and a successful connection was achieved at the first attempt. The local IT team used an Integration Engine (BizTalk) to process all messages between local systems and the Quality Compass system. This worked successfully and meant that there were no direct links between Quality Compass and the Trust s systems. Post Installation The exchange of data between the local system and Quality Compass was across a secure FTP connection and was 100% stable with no timeouts. However, it proved impossible during the course of the project to effectively integrate information from the Trust s PAS and microbiology laboratory system. Details of the issues which arose after Quality Compass had been installed and when the Trust s IPT started to use the system are set out in Table 2, along with the action that was taken to resolve them, and the outcome of that action. 4 HL7 is a not-for-profit, American National Standards Institute accredited standards developing organization dedicated to providing a comprehensive framework and related standards for the exchange, integration, sharing, and retrieval of electronic health information that supports clinical practice and the management, delivery and evaluation of health services. 7

Following the completion of the evaluation, Vecna Technologies Inc identified areas of development in relation to each issue and this is included in Table 2. Issue Action Outcome Vecna Response 2.1 The Trust s IT team had understood from discussions with ABC that the system was fully customisable, but this understanding turned out to be incorrect. 2.2 There was an issue with patient information coming through on Quality Compass without a hospital ID number or an NHS number. For example, a All requests for customisation turned out to be impossible to meet. For example, when designating infections as community acquired or hospital acquired the user had to do this without interrogating the information e.g. the user wanted to review the background information first in order to decide where to attribute the infection/result, but was unable to do so. Responsibility for resolving this issue was debated between ABC and the local IT team, and ABC eventually agreed to carry out the work to resolve the problem. Note that patient identifier is one of the required 8 Customisations are managed through a product development process. The request by the Trust to review all patient data in aggregate was fulfilled in the product s subsequent Version 5.1 release which was not available for evaluation during this project. ABC added searches for NHS number, in the absence of this number a search for the sample ID number (laboratory specimen number) was Vecna has provided customisations for large health systems with specific needs and can provide this as a service to NHS Hospital Trusts. This service was not included as part of the Advisory Board Quality Compass Solution. As an example, Vecna has a relationship with the U.S. Department of Veterans Affairs and has performed significant custom feature development. Vecna confirms that QC PathFinder 5.1 has the capability to aggregate and unify patient data for review prior to designation. Several modifications were made to the patient identifier mapping. It is a prerequisite for the system to function, and Vecna can work with the vendor of the source

sample sent in from a GP. 2.3 In the absence of information regarding the location of the patient, the system used the last known location of the patient. This meant, for example, that patients who had been tested in the community might be assigned by the system to the ward they had been in during their last hospital admission. 2.4 The Microbiology laboratory system was unable to filter out GP requests and requests from the local rehabilitation hospital. fields. Absence of this information limits data aggregation functionality and requires work-around measures. A solution was developed by the local IT team to match the sample request date from the lab results to the ADT messages based on best fit of dates. A process was developed to try and alleviate this problem. made. It is important to note that this information is a prerequisite for the system to function as expected. The newer, untested Version 5.1 release allows the user to override the default (and potentially inaccurate) location of attribution. The process was not deemed scientifically accurate. Consequently, the IPT did not trust the information in the Quality Compass system and continued to use their existing systems as a check. system to help identify or provide the necessary exports for the data in question. This is a model that Vecna has utilised in the past to ensure the highest quality data for QC PathFinder. Vecna confirms that the Version 5.1 of QC PathFinder allows manual designation of the infection s location of attribution. Vecna can create a white list of locations from which laboratory results should be used by the system. Vecna can work with health care administrators to create a white list to result in more accurate filtering of GP and local rehab requests. 9

2.5 The workaround processes required matching lab request dates with the Trust s PAS data across millions of ADT events. This was very processor and memory intensive and tended to put the Trust server under a considerable load. As a consequence there were occasions when the next batch of matching commenced before the previous one had finished which resulted in empty files being forwarded to ABC. Unfortunately, due to the mechanism to extract and format data, there was no way to identify any problems during this matching process. Therefore, the only way the IT team were aware of a problem was when the IPT reported no data was available The data was always reprocessed and resent as a priority. There was no technical solution afforded, thus, the process continued in this way. Vecna can handle the matching process on the hosted side of the solution rather than at the source system level. This would result in less server load and would allow more efficient processing within a robust data centre. 10

from Quality Compass - for example, no alerts. 2.6 Although the technical specifications were clear, it transpired that the full requirements which would need to be met if the product was to work had not been made clear by ABC prior to commencement of installation. For example an episode number was required to be passed from the laboratory feed in order to correlate the corresponding Admission, Discharge and Transfer (ADT) messages and lab results and thus identify where the test was undertaken. There was no capacity on the lab system to harvest or output episode numbers. 2.7 Quality Compass gathered information on where samples were collected A solution was developed by the local IT team to match the sample request date from the lab results to the ADT messages based on best fit of dates. Note that encounter/episode number is a required field for Quality Compass. Absence of this information limits data aggregation functionality and requires work-around measures this does not form part of the technical specification and it is suggested that trusts ensure they are fully aware of the real requirements in the data they import. This issue was referred to ABC and the local IT team. GP and local rehabilitation hospital results did not This workaround process was not believed to be 100% accurate because patients may have several ward movements on any given day. Consequently the IPT did not trust the information in the Quality Compass system and continued to use their existing systems as a check. This issue was not resolved. Vecna s standard process for implementation includes the delivery of an Interface Control Document that specifically defines what data will be required, what data will be sourced from the client s systems, and the risks for not providing data elements. This would be delivered at the onset of a project for a direct Vecna implementation. Vecna can work with the vendor of the source system to determine the proper mapping for a visit or episode number to correlate ADT and lab results. Another option would be to work with the IPT to create a mapping workaround that is more accurate, using validation test sets as necessary. Vecna can define custom business rules to ensure that appropriate data coming from the GP and local 11

from PAS files rather than from the Microbiology laboratory system. As a result, information was not provided where the source of the sample was a GP or the local rehabilitation hospital. Potentially, this could have led to certain information usually generated by the IPT (e.g. MRSA or Clostridium difficile notification letters) not being sent, though existing local processes in place prevented any such occurrences. 2.8 The system was not able to show at a glance whether information had already been viewed and acted on, thus running the risk of actions being duplicated by different members of staff and hence time wasted. It was suggested that a comments field should be included. contain the same patient identifiers as the other systems (which is to be expected), which is a required field in all parts despite the system claims that connectivity to all elements of electronic patient data capture are possible.. Absence of this information limits data aggregation functionality and requires work-around measures. This issue was raised with ABC. ABC informed the Trust that software development had been finalised for two further releases and consequently this suggestion could not be implemented during the evaluation. This had a negative impact on the use of the system as the IPT did not want rehab hospitals does come through the system. In particular, this would be an enhancement extending the white list to include appropriate situations where data from GP and local rehab hospitals would not be filtered out. The current version of QC PathFinder describes in the line-listing whether infections have been clinically confirmed by a user. In addition, an upcoming release of QC PathFinder will allow users to document interventions performed upon patients. A comments field is 12

2.9 Quality Compass has the facility to access recorded archived teleconferences but when staff at the Trust tried to utilise this facility, they experienced technical problems arising from the sound capabilities of their personal computers. This issue was referred to the local IT team. to duplicate work, such as phoning a result through to a ward if someone else from IP had already undertaken that task. This issue was not resolved as the cause of the problem was never diagnosed. available within the infection designation screen. The archived teleconferences are specifically in the scope of ABC services. Table 2: Issues that arose following installation of the system How acceptable was the product to the Infection Prevention Team? As a result of the issues set out above, the views of the IPT were inevitably affected by the failure of Quality Compass to deliver accurate information consistently. This meant that they had to check all information coming from Quality Compass against that coming from existing systems, in order to allow them to ensure that the Quality Compass system was picking up all of the cases that the existing system was detecting. The period over which Quality Compass was used also coincided with a period of extreme staff shortages in the team, which led to less utilisation of the system than expected, and a small number of comments being received. The team s view, however, was that if the issue of reliability could be overcome, they could see the potential of Quality Compass, which they found easy to use. 5 members of the team had received training from ABC, with 3 describing it as adequate and 2 as excellent, whilst 2 received training from a colleague at the trust. 13

One of the key components of the Quality Compass system is its ability to provide access to the Quality Compass Cohort of users and research professionals expert in the field of epidemiology. This however does not appear to be an aspect of the system that was utilised to any extent during the course of the trial. As a result of the cohort of users and research professionals being located at hospitals in the US and due to the time differences between the UK and the US it was not a viable option for staff at the Trust to participate in these teleconferences. ABC has stated that they plan to introduce this service in UK working hours amongst UK hospital cohorts. Advice for trusts considering introducing Quality Compass Important Points to Consider Any Trust considering introducing Quality Compass should consider carefully what they hope to achieve by doing so, and explore in some depth whether it is likely to be capable of meeting their needs, bearing in mind the capabilities of the Trust s own IT systems. It is useful to draw up a list of user requirements prior to approaching any surveillance system company to ensure the system meets trust requirements exactly. IT The trust s own IT department must be able to assign a contact member of staff to liaise with ABC as well as with the end users within the trust. This is seen as vital in the set up phase of the implementation but must be maintained throughout the use of the system. The roles and responsibilities of this person are suggested as follows: Act as contact person for technical aspects of the system between the Trust and ABC Input into original contract and data specifications taking into account the needs of the end users Generation of and performance of user acceptance testing prior to go live Ensure the systems to be linked to, such as microbiology laboratory system, are capable of an output which can be merged directly with the QC system. Care needs to be taken in advance to ensure key data fields are both present and in the correct format for use and this includes carespell and/or episode numbers for example, without which the data cannot be transferred. Ensure the availability and use of fallback systems until the ability of Quality Compass to meet the essential requirements of the Trust has been clearly established. 14

Review and acceptance of the security of the ABC data servers on behalf of the Trust and according to trust policies and UK law including governance documentation. Review end user system specifications and ensure access for all to the ABC Quality Compass Cohort of users and research professionals expert in the field of epidemiology. This can take the format of archived teleconferences (to circumvent time difference with the US) but sound cards and internet access are key. Set up and maintain a secure FTP link. System Amendments It must be made clear what the Trust s expectations are in terms of updates to the system, the scope, and the timescale for any requested amendments must be recorded and agreed. Wherever possible the IPT must work closely with both ABC and local IT to ensure the system will contain the data they need to perform their roles successfully. It is important to note that the system at the time of the evaluation was not customisable. Responsibilities It will be important to be clear where responsibility for addressing problems will lie. Staffing The Trust must ensure that sufficient staff have full access to the system in order to work effectively. Where evaluation is required, it is likely to be impacted if any particular team is under resourced during that time. ABC has staff available for face-to-face training in the UK. However, this has to be arranged in advance as staff travel from the US. It has been seen in interactions with other companies in this field that having a team on the ground does lead to improved relationships and a greater understanding. Fallback It will be important to ensure the availability and use of fallback systems until the ability of Quality Compass to meet the essential requirements of the Trust has been clearly established. However, this needs to be balanced with the need at some point to withdraw the availability of existing systems, in order to ensure that Quality Compass is used. 15

Post evaluation Upon commencement of the evaluation, the Trust contracted the services of the ABC and not Vecna Technologies Inc directly, which is now something the company can offer to the NHS. Since the evaluation was conducted, Vecna has released three new versions of QC PathFinder and claim to address the product issues described in this report. Vecna is now able to provide advanced systems integration services required by the NHS through a direct relationship with NHS Hospital Trusts, resolving potential issues faster. This has not been validated by the authors of this report. 16