White Paper HIGH AVAILABILITY CONFIGURATION FOR HEALTHCARE INTEGRATION PORTFOLIO (HIP) REGISTRY EMC Documentum HIP, EMC Documentum xdb, Microsoft Windows 2012 High availability for EMC Documentum xdb Automated recovery ECD Healthcare Abstract This white paper describes a solution that offers high availability for EMC Documentum xdb document sharing as part of the Healthcare Integration Portfolio product suite. The paper also describes how to determine the functionality of the components within the virtualized solution. July 2015
Copyright 2015 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. All information included in this document is provided "as-is" and does not guarantee EMC's support or approval. Please consult the official product documentation or the Product Support team regarding any questions of supportability. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H14229 2
Table of contents Executive summary... 5 Business case... 5 Solution overview... 6 Key results/ recommendations... 6 Introduction... 7 Purpose... 7 Scope... 7 Audience... 7 Terminology... 7 Technology overview... 8 Overview... 8 EMC Documentum HIP components... 8 EMC Documentum xdb... 8 VMware vsphere... 8 Microsoft Server Failover Clustering... 9 Architectural overview... 10 Physical environment... 10 Hardware resources... 11 Software resources... 11 VMware datastores... 11 Virtual configuration... 13 ESX configuration... 13 VMware network... 13 Virtual machine resources and software... 13 Design considerations... 15 DNS configuration... 15 Microsoft Server Failover Clustering... 15 xdb Master (Read Write) Cluster configuration... 16 xdb Replica (Read Only) Cluster configuration... 19 Test results... 23 Overview... 23 Test objectives... 23 Results and observations... 24 xdb Read Write master failover... 24 3
xdb replica failover... 27 Conclusion... 29 Summary... 29 Findings... 29 References... 30 Product documentation... 30 4
Executive summary This white paper describes the high availability (HA) solution for the Healthcare Integration Portfolio (HIP) Registry using EMC Documentum xdb, Microsoft Server Failover Clustering, VMware vsphere, and Microsoft Windows Server. The purpose of this white paper is to demonstrate the automated failover capability of EMC Documentum xdb using Microsoft Server Failover Clustering. This solution provides a cross-document sharing environment including: EMC Healthcare Integration Portfolio (HIP) EMC Documentum xdb Microsoft Server Failover Clustering VMware vcenter We 1 built and tested this solution on a VMware virtualized infrastructure. The architecture used for testing provides: Guidelines to build similar solutions A high availability procedure using Microsoft Failover Clustering Business case Healthcare IT management is challenged by the growing volumes of medical imaging data that result from: Technological advances in modalities Increased number of diagnostic studies and number of images per study Lengthening of diagnostic image retention requirements Compounding these challenges is the fact that healthcare IT management also faces the difficulty of integrating and being able to easily share unstructured data with other patient data and presenting this data to the caregiver at the point-of-decisionmaking anywhere in the enterprise. Unstructured data includes medical images from departmental Picture Archiving and Communication Systems (PACS), such as radiology, cardiology, oncology, pathology, and mammography, for which healthcare organizations need to make informed decisions at the point of care. To provide optimum patient-care delivery, IT managers are investing in automated tools to integrate the stovepipes of information created by numerous proprietary applications, and to efficiently manage the increasingly complex infrastructure of both physical and virtual assets in their environments. To move toward digital-care processes, healthcare organizations typically move through multiple investment phases, such as clinical workflow processes. They must change in parallel with IT investments, including adoption of and adherence to industry standards as well as information digitization, sharing, and management. 1 The use of we refers to the team of EMC and partner solutions engineers who designed, built, and validated the solution. 5
To support these investments, healthcare providers need to address the following: Increasing amounts of storage required to support legacy and new healthcare applications Increasing costs and operational impact of maintaining different PACS infrastructures Adopting DICOM and HL7 standards that facilitate the sharing of images and critical information associated with them Providing a higher level of transparency and information access to patient data and medical images Centralizing, securing, and managing data storage As healthcare systems focus more on collaborative patient care and the continuity of care delivered through a network of healthcare providers operating out of traditional settings, the need for flexible and cost-effective information systems is more critical. Your organization needs collaboration across various settings, but traditional IT infrastructures cannot always effectively support this due to cost issues, long development time, and a lack of flexibility. Solution overview Key results/ recommendations This solution provides high availability for EMC Documentum xdb systems and shows the ability of Microsoft Failover Clustering to automate the failover and recovery processes. This solution enables customers to: Provide high-availability for EMC Documentum xdb systems Automate the failover of the xdb Master and Replica Server to a standby Master and Replica Server using Failover Clustering Access clinical documents with read/write privileges in the event of a failure of the xdb database The testing of this solution enables you to understand the requirements to create a high-availability solution for xdb. It provides you with the background to explain the recovery time and procedures needed for the environment to be automatically brought back online when the xdb servers experience an outage. 6
Introduction Purpose Scope Audience Terminology The purposes of this white paper are to validate: The test methodology used to determine the functionality of the highavailability HIP solution xdb Server automated high availability using Microsoft Failover Clustering The scope of this white paper includes: Measuring functionality of Microsoft Failover Clustering recovery technology Documenting the process of automated xdb master and replica recovery using Microsoft Failover Clustering This white paper is intended for EMC employees, partners, and customers, including IT planners, xdb Server database administrators (DBAs), and EMC field personnel who are tasked with deploying such a solution in a customer environment. It is assumed that the reader is familiar with the various components of the solution. This white paper includes the terminology shown in Table 1. Table 1. Terminology Term DICOM HIP Definition Digital Imaging and Communications in Medicine Healthcare Integration Portfolio HL7 Health Level 7 IHE PACS RIS WDK xdb XDS Integrating Healthcare Enterprise Picture Archiving and Communication System Radiology Information System Web Development Kit Documentum XML database Cross-Enterprise Document Sharing 7
Technology overview Overview This section provides an overview of the primary technologies included in this solution: EMC Documentum HIP components: XDS Registry XDS Repository EMC Documentum xdb VMware vsphere Microsoft Failover Clustering EMC Documentum HIP components The HIP XDS Registry is the directory or white pages for medical and administrative content that provides applications within the enterprise with the ability to discover and then access information. Based on the implementation of Integrating Healthcare Enterprise s (IHE) XDS Registry specifications, it provides a central catalog for documents that reside in a federated system or in repositories, either heterogeneous or geographically distributed. The HIP XDS Repository stores structured and unstructured healthcare information where all patient-centric documents, images, and media are available via the XDSspecification for applications to consume even when the clinical, financial and operational content was not created via an XDS specification. EMC Documentum xdb EMC Documentum xdb is a native XML database. Its scalable architecture and complete support for the XQuery language enables any organization to warehouse content in an application-neutral format. These databases do not depend on a proprietary application for information retrieval. Unlike relational databases, xdb enables you to easily modify content schemas to adapt to changing information requirements and supports queries against complex data structures that are not easily modeled in rows and columns. In addition, with its powerful and extensible development and runtime toolset, xdb is a powerful platform for the most complex and demanding content-centric applications. VMware vsphere VMware vsphere uses the power of virtualization to transform data centers into simplified cloud computing infrastructures and enables IT organizations to deliver flexible and reliable IT services. vsphere virtualizes and aggregates the underlying physical hardware resources across multiple systems and provides virtual resource pools to the data center. As a cloud operating system, vsphere manages large infrastructure collections (such as CPUs, storage, and networking) as a seamless and dynamic operating environment. It also manages the complex operations of a data center. 8
Microsoft Server Failover Clustering Microsoft Server Failover Clustering monitors and automates the restart process of your application and data services on an alternate server for Microsoft Windows environments. Failover Clustering also automates the failback of your services, applications, and data managing both failover directions and ensuring consistent and error-free business continuity. 9
Architectural overview This white paper characterizes and validates high availability for a virtualized EMC Documentum xdb and XDS environment using Windows 2012 R2 and Microsoft Server Failover Clustering. VMware is used to provide the virtualization layer. Physical environment Figure 1 shows the physical environment. Figure 1. Physical environment 10
Hardware resources Table 1 shows the hardware resources used in this solution. Table 1. Hardware resources Equipment Quantity Configuration Server 2 128 GB memory servers with 2.27 GHz, 8C processors VNX5500 storage array 1 75 x 600 GB SAS 10k drives Fibre Channel (FC) switch 2 8 GB, 80 ports Network switch 1 10 GB, 24 ports 75 x 2 TB NL-SAS 7.2k drives Software resources Table 2 shows the software resources used in this solution. Table 2. Software resources Software Version Notes EMC VNX5500 VNX OE for Block FLARE release 32 Operating environment for the block VMware ESX 5.1 Update 1 Server hypervisor EMC PowerPath Virtual Edition 5.9 Multipathing and load balancing for block access VMware vcenter Server 5.1 Update 1 vsphere Management Server Microsoft Windows Server 2012 R2 Operating system for all virtual machines EMC Documentum xdb 10.5.9 Installed on Microsoft Windows. Includes xdb Master and xdb Replica EMC Documentum HIP components 1.7 Installed on Microsoft Windows. Includes XDS Registry and Repository EMC Documentum 7.2 Used for hosting Documentum Repositories VMware datastores As shown in Table 3, we created a 70 GB VMFS datastore per virtual machine to house each of the virtual machines. We also created an individual VMware Virtual Machine File System (VMFS) datastore for each of the databases and log disks. 11
Table 3. VMWare Datastore configuration details Virtual machine Size/GB Type Datastore Drive letter xdb Master 70 VMFS xdb Master C:\ 48 VMFS xdb database D:\ 18 VMFS xdb Log L:\ xdb Master Standby 70 VMFS xdb Master Standby C:\ xdb Replica 70 VMFS xdb Replica C:\ 48 VMFS xdb Replica database D:\ 18 VMFS xdb Replica log L:\ xdb Replica Standby 70 VMFS xdb Replica Standby C:\ XDS Registry 70 VMFS XDS Registry C:\ XDS Repository 70 VMFS XDS Repository C:\ Documentum 70 VMFS DCTM node C:\ Documentum file store 200 VMFS Filestore1 E:\ 12
Virtual configuration ESX configuration VMware network Virtual machine resources and software The virtual infrastructure consisted of two servers to form a VMware ESX HA cluster with version 5.1 installed. We configured all virtual machines to use VMXNET3 adapters. This included a 10 GbE backbone network connecting the physical ESX servers. We connected a virtual distributed switch for the ESX cluster into which the 10 Gb ports were added. Table 4 lists the virtual resources allocated to each of the virtual machines and the software we installed on each machine. Table 4. Virtual machine allocations Virtual machine Quantity Software Documentum Content Server and Docbroker Documentum XDS Registry 1 Microsoft Windows Server 2012 R2 Apache Tomcat 7 Microsoft SQL Server 2012 Client Java JDK 1.7.0_67 Documentum Content Server 7.2 containing: XDS Repository Content Storage Services enabled Retention Policy Services enabled Trusted Content Services enabled Documentum Docbroker Solution custom DAR file installed to XDS Repository VMware tools 1 Microsoft Windows Server 2012 R2 Java JDK 1.7.0_67 Documentum HIP Registry 1.7 Tomcat 7 VMware tools XDS Repository 1 Microsoft Windows Server 2012 R2 Java JDK 1.7.0_67 HIP Repository 1.7 Tomcat 7 VMware tools 13
Virtual machine Quantity Software xdb Master (Read Write) Server xdb Replica (Read Only) server 2 Microsoft Windows Server 2012 R2 Java JDK 1.7.0_67 Documentum xdb 10.5.9 Tomcat 7 VMware tools Microsoft Windows Server Failover Clustering 2 Microsoft Windows Server 2012 R2 Java JDK 1.7.0_67 Documentum xdb 10.5.9 Tomcat 7 VMware tools Microsoft Windows Server Failover Clustering 14
Design considerations We designed the xdb Master Read Write Servers to share the VMFS database and log disks, as shown in Figure 2, so that the standby server could seamlessly take over the function of the Master Server in the event of a failure. The xdb application was installed on the master node with the database and logs placed on the respective drives. We then installed xdb on the Master Standby Server. We designed the xdb Read Only Servers in the same way with their own database and log disks. Figure 2. xdb server disk configuration DNS configuration We created a DNS alias for the xdb Master Server called W2K12xDBRWClus. This alias was then used for all communication between the Read Write Cluster and Read Only Clusters. Microsoft Server Failover Clustering Microsoft Server Failover Clustering monitors and automates the restart process of your application and data services on an alternate server for Microsoft Windows environments. Failover Clustering also automates the failback of your services, applications, and data managing both failover directions and ensuring consistent and error-free business continuity. 15
xdb Master (Read Write) Cluster configuration After xdb has been installed on the master node(s) the EMC xdb service needs to be set to manual so that the Microsoft Failover Cluster Service can manage the service The xdb Read Write Cluster was configured with the EMC xdb Server service as shown in Figure 3. The EMC xdb server role was configured using a generic service type. The configuration also contained the xdb master and xdb master standby virtual machines, as well as the following configuration settings: Figure 3. xdb server service 16
Figure 4 shows the Node list in xdb master (Read Write) Microsoft Failover Cluster. Figure 4. xdb master (Read Write) Cluster Node Configuration Figure 5 shows the disks created in Microsoft Failover clustering. Figure 5. xdb master (Read Write) Cluster disks 17
Figure 6 and Figure 7show the xdb master login screen and main screen prior to failure. Figure 6. xdb Master (Read Write) login screen Figure 7. xdb Master (Read Write) main screen 18
xdb Replica (Read Only) Cluster configuration After xdb has been installed on the replica node(s) the EMC xdb service needs to be set to manual so that the Microsoft Failover cluster service can manage the service Figure 8 shows the script that needs to be run on one of the replica nodes, in each cluster, once during its lifetime. This is run after the xdb software has been deployed on both cluster nodes. The reason why this script is required is because in xdb, a replica federation is created by copying a master federation. After starting a replica server on the replica federation, the replica server keeps in sync with the master by replaying transaction logs that come from the master. The master federation knows about the replica because a replica id has been registered. The master server keeps obsolete log records it would otherwise delete, until the replica server ask for them and confirms it got them. One of the unwanted effects of creating a replica is that the replica on its turn has the same replica id registered and on its turn keeps otherwise obsolete log records until yet another replica would ask for its log records call xdb.bat --debug -y create-replica --replicabootstrappath d:\\data\\xhivedatabase.bootstrap --replicaid 10 --federation xhive://w2k12xdbrwclus:1235 --password xxxxxxx start /B xdb.bat --debug -y run-server --master xhive://w2k12xdbrwclus:1235 --webserver-port 0 --address 0 -- federation d:\\data\\xhivedatabase.bootstrap --port 1235 -- replicator 10 call xdb.bat --debug -y create-replica --remove 10 --federation xhive://w2k12xdbroclus:1235 --password xxxxxxx call xdb.bat --debug -y stop-server --federation xhive://w2k12xdbroclus:1235 --password xxxxxxx Figure 8. xdb Read only create replica script Figure 9 shows the run script that brings the xdb application online on the xdb Replica Read Only cluster. The create script above needs to be executed before the run script is configured in the cluster configuration. This ensures that the server is synchronized with the xdb master Read Write server. We connected to the xdb alias, W2K12xDBRWClus, to ensure all updates were synchronized to the replica. xdb run-server --master xhive://w2k12xdbrwclus:1235 -- replicator 10 --address 0 --federation d:\\data\\xhivedatabase.bootstrap Figure 9. xdb Read only run server script 19
Figure 10 and Figure 11 show the configuration of the EMC xdb Server service within the xdb Replica (Read Only) Cluster. The xdb Read only role was configured as a generic application with the run server script shown in Figure 11. The configuration contained the xdb replica and xdb replica standby virtual machines as well as the disks Figure 10. xdb Replica (Read Only) Cluster Configuration 20
Figure 11. xdb Replica (Read Only) Cluster Configuration Figure 12 shows the node list in xdb Read Only Microsoft Failover Cluster. Figure 12. xdb Replica (Read Only) Cluster Node Configuration 21
Figure 13 shows the disks that we used in xdb Read Only Microsoft Failover cluster. Figure 13. xdb Replica (Read Only) Cluster disks Figure 14 shows the xdb Read Only Replica Server login screen, with the W2K12xDBRWClus, replication master. Figure 14. xdb Replica (Read Only) login screen 22
Test results Overview Test objectives This chapter documents the results and observations made during the testing of high availability for Documentum HIP xdb. The objectives of the test were to initiate a failure of the xdb Master Read Write and xdb Read Only Replica Server during clinical document ingestion and observe the recovery process. To validate the environment we executed the ProvideandRegister Registry call to ingest clinical data into the XDS Registry. All the testing was based on a single site implementation of EMC Documentum Healthcare Information Portfolio (HIP) and EMC Documentum xdb. These tests did not include the failover of XDS Registry or XDS repository. For more information on the failover of these products refer to the following document: EMC Medical Image Management with Document Sharing Solution (Enterprise) Business Continuance 23
Results and observations xdb Read Write master failover Figure 15 shows the Microsoft Failover interface during a failure of the xdb master node. Figure 15. Microsoft Failover Interface showing xdb Master (Read Write) failover 24
Figure 16 shows the Microsoft Failover cluster Interface after a failover. You can see that the resources are now located on the xdb master standby node W2K12XDB-2. The total time to fail over from the master to the standby was approximately 15 seconds. The Replica Server automatically reconnects to the Master Server for updates. Figure 16. Microsoft Failover Interface showing xdb Master (Read Write) after failover 25
Figure 17and Figure 18 show the xdb master login screen and the main screen after failover. Figure 17. xdb Master (Read Write) master login screen Figure 18. xdb Master (Read Write) master main screen 26
xdb replica failover Figure 19 shows the Microsoft Failover Cluster interface during a failure of the xdb replica node. Figure 19. Microsoft Failover console during xdb Replica (Read only) failover Figure 20 shows the xdb replica login screen after failover. The replica receives updates from the xdb Master Server W2K12xDBRWClus. The Replica Server automatically reconnects to the Master Server for updates. The total time for failover of the replica is ten seconds. 27
Figure 20. xdb Replica (Read only) login screen after failover 28
Conclusion Summary Findings The testing emulated failure scenarios within a Cross-Enterprise Document Sharing environment and shows that the solution provides the following benefits: High availability for EMC Documentum xdb using Microsoft Failover Clustering The use of standby servers for the xdb Read Write master and xdb Read Only replica provides resilience The test results demonstrate that this solution provides: xdb master failover in approximately fifteen seconds xdb replica failover in approximately ten seconds Automatic reconnection of the xdb replica to the xdb master 29
References Product documentation For additional information, see EMC.com and the product documents listed below. Industry Solutions on EMC.com Choosing an xdb Configuration EMC Medical Image Management with Document Sharing Solution (Enterprise) Business Continuance 30