TECHNICAL BRIEF: DATACANE........................................ Datacane - When the bits hit the fan! Who should read this paper This paper is intended for those who have seen the Symantec Datacane video and wish to know more about the underlying technology used in its production.
Content Introduction........................................................................................................... 1 Overview.............................................................................................................. 1 The Failover Scenario................................................................................................... 4 Configuration Details................................................................................................... 6 Appendix 1 - Server Rack Build Details................................................................................... 6 Appendix 2 - Primary Database Cluster Configuration..................................................................... 11 Appendix 3 - Primary Application Cluster Configuration................................................................... 17 Appendix 4 - First Primary Web ApplicationHA Configuration.............................................................. 25 Appendix 5 - Second Primary Web ApplicationHA Configuration............................................................ 27 Appendix 6 - Recovery Site Single Node DB Cluster Configuration.......................................................... 29 Appendix 7 - Recovery Site Single Node App Cluster Configuration......................................................... 33 Appendix 8 - Recovery Site Web Server ApplicationHA Configuration....................................................... 40 Appendix 9 - Primary Database Server Volume Manager (VxVM) Configuration............................................... 42 Appendix 10 - Recovery Site Database Server Volume Manager (VxVM) Configuration........................................ 46 Appendix 11 - Primary Application Server Volume Manager (VxVM) Configuration............................................ 49 Appendix 12 - Flexible Storage Sharing Configuration..................................................................... 51 Appendix 13 - Stills From The Datacane Video............................................................................ 52
Introduction This paper describes the technical architecture of the hardware and software components used in the Symantec Information Availability team's "Datacane" video production. The "Datacane" video was produced in September, 2013 by Symantec product marketing to promote Symantec Information Availability products. It dramatizes a scenario in which an organization, threatened by an approaching hurricane must migrate its most mission critical applications to a remote data center just minutes before a storm surge is forecast to occur and possibly flood the data center. The Symantec solutions used and their role in the video are: Symantec Storage Foundation Cluster File System: Provides cluster volume management and cluster file system services to servers in a cluster, allowing all servers to concurrently access the cluster file system data. Symantec Cluster Server: Provides local high availability and wide-area disaster recovery to applications running in physical or virtual machines by automating the start, stop, monitoring and failover of the application. Symantec Application HA: Provides high availability for applications running in a virtual machine by automating the start, stop, monitoring and in-place restart of the application. Symantec Replicator: Provides continuous volume replication and periodic file replication between sites for disaster recovery purposes. Veritas Operations Manager: Provides centralized, "single pane of glass" management and visibility into the Storage Foundation, Cluster Server and Application HA configurations. Overview The server rack designated as the primary data center was situated in the Insurance Institute for Business and Home Safety's research center in Richburg, South Carolina while the rack designated as the secondary, or recovery data center, was placed in a data center co-location facility in Ashburn, Virginia. Both "data centers" and the migration of the applications were monitored from Symantec's Executive Briefing Center in Mountain View, California. The IBHS research center was built to measure the effect of extreme weather conditions on structures and other property, and was used for the production of the "Datacane" video to simulate hurricane-force winds and the accompanying storm surge. Figure 1: The IBHS research center in Richburg, South Carolina 1
Architecture Summary Primary Site The primary data center located in Richburg, SC consisted of two physical servers running VMware vsphere 5.1. Each server hosted a set of virtual machines, some providing local management services such as VMware vcenter, Veritas Operations Manager (VOM), Symantec HA console services and DNS. The remaining virtual machines hosted the database, application and web tiers of a multi-tier application stack mimicking Symantec's Norton e-store architecture, as well as idle virtual machines to provide local high availability in the event of an isolated local failure. All the database and application virtual machines at the primary site had Symantec Storage Foundation Cluster File System High Availability installed. This solution bundle provides storage management through the Cluster Volume Manager and Cluster File System extensions to Symantec Storage Foundation, allowing file systems to be mounted concurrently across all cluster nodes. It also includes Symantec Cluster Server to provide local high availability and wide-area disaster recovery services. The web servers had Symantec ApplicationHA installed, which provide application start, stop, monitoring and restart within a virtual machine. (Please refer to Figure 2 for a logical block diagram of the configuration.) At the primary site, virtual machines were configured as a 2-node cluster for the Oracle database server, a 2-node cluster for the Oracle WebLogic application server, and the Apache web servers were configured as two standalone virtual machines with the web server instances managed by Symantec ApplicationHA. Veritas Operations Manager was employed to manage all storage and HA/DR for the site, and to configure Virtual Business Services which manages mulit-tier application dependencies. This allows the automated startup, shutdown and failover of multi-tier application stacks like the Norton e-store to be properly orchestrated and ensures all application components are started and stopped in the proper sequence. With the latest (6.1) version of Symantec Storage Foundation Cluster File System installed, it was possible to employ a "shared nothing" storage model consisting of 100% direct-attached storage (DAS) on all nodes, using the new Flexible Storage Sharing feature. This negated the need for any complex SAN infrastructure as storage directly attached to one host was presented to the other node in the cluster across the cluster interconnects. Architecture Summary Recovery Site The recovery data center located in Ashburn, VA consisted of a single physical server running VMware vsphere 5.1. Similar to the primary data center, this server hosted virtual machines providing local management services as well as virtual machines for all three tiers of the Norton e-store application stack. There was no second physical server or idle virtual machines for local high availability, which is a common approach to disaster recovery site configuration. (Please refer to Figure 2 for a logical block diagram of the configuration.) The database and application virtual machines each had Symantec Storage Foundation High Availability installed, which provides nonclustered storage management as well as Symantec Cluster Server for high availability and disaster recovery. The web server virtual machine had Symantec ApplicationHA installed. As with the primary site, Veritas Operations Manager was employed here to manage storage, HA/DR and Virtual Business Services. The application's database files were replicated from Richburg to Ashburn using Symantec Volume Replicator. Symantec Volume Replicator is a block-level continuous data replication solution, and was set in this configuration to replicate asynchronously due to the distance involved. 2
Figure 2: Logical diagram of the disaster recovery architecture Figure 3: Detail of virtual machine builds. Note that at the primary site (Richburg), there were two database and application server virtual machines for redundancy 3
The Failover Scenario With the Norton e-store service running at the primary data center in Richburg and its data being continuously replicated to the recovery site in Ashburn, Symantec employees monitored application status from consoles in both Richburg and Symantec corporate headquarters in Mountain View, California. With a hurricane approaching, the employees were alerted to the likelihood of a storm surge only minutes away. From the Veritas Operations Manager console, it was confirmed that the Virtual Business Service named Norton Primary was currently running: Figure 4: View of Norton e-store service at primary site prior to migration Having made the decision to migrate this mission critical service to the recovery site 1, operations staff selected from the VOM console the appropriate recovery plan, and executed it: 1- Note that this type of failover differs from a failover that occurs automatically as the result of a failure. In this instance, the failover is initiated in anticipation of a likely disaster, and is preferable since it allows for a graceful shutdown of the applications and orderly restart at the recovery site. The Symantec solution set works the same way in either scenario, though. 4
Figure 5: Executing a recovery plan Once failover was initiated, the storm surge was only three minutes away. At the three minute mark, the storm surge waters flooded the primary data center, and employees in Mountain View, monitoring critical services from their Veritas Operations Manager console, confirmed that the Norton e-store service was fully recovered and available. 5
Configuration Details The appendices that follow contain the details of the server and cluster configurations for each site and system. While not strictly a reference architecture, it gives a very detailed description of how a multi-tiered application stack such as the Norton e-store might be configured for disaster recovery using Symantec solutions. It's assumed that the reader is familiar with both Volume Manager and Symantec Cluster Server configurations and output. Appendix 1 - Server Rack Build Details Briefly, the Primary Data Center and Recovery Data Center configurations were built on Dell R720 servers running VMware vsphere 5.1. The virtual machines configured to run the various tiers of the multi-tier application stack shown in the video were all Red Hat Enterprise Linux 6.x systems running Symantec Storage Foundation/HA version 6.1. Primary Site (Richburg, SC): ESXi 5.1 build: 64 GB RAM 2 x Intel Xeon E5-2620 processors (6 core processors, with Hyperthreading capabilities, so VMware sees it as 24 logical processors) Mellanox ConnectX-3 Dual Port: Both ports VM DirectPath to Oracle VM 300 GB SAS Disk #1: 10 GB ESXi partition, remainder VMFS datastore1 300 GB SAS Disk #2: VMFS datastore2 300 GB SAS Disk #3: VMFS datastore3 400 GB SSD: VM DirectPath to Oracle VM ESXi Host #1 Guest Virtual Machines: vcenter 5.1 Windows 2008R2 8 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 50 GB VMDK on datastore2 (OS) Symantec HA Console Windows 2008R2 4 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 40 GB VMDK on datastore2 (OS) Web Tier Red Hat Enterprise Linux 6.X 6
4 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 24 GB VMDK on datastore2 (OS) 4 GB swap 20 GB / (ext4) Application Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) NIC3 on vswitch3 (VCS HB1) NIC4 on vswitch4 (VCS HB2) 80 GB VMDK on datastore3 (CFS/FSS, Application data) 32 GB VMDK on datastore2 (OS) 8 GB swap 24 GB / (ext4, room for application binaries) Database Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) VM DirectPath Infiniband Port1 (VCS HB1) VM DirectPath Infiniband Port2 (VCS HB2) VM DirectPath SSD (CFS/FSS, Oracle data) 32 GB VMDK on datastore2 (OS) 8 GB swap 24 GB / (ext4, room for Oracle binaries) ESXi Host #2 Guest Virtual Machines: Active Directory/DNS Windows 2008R2 2 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 7
40 GB VMDK on datastore2 (OS) Veritas Operations Manager (VOM) Red Hat Enterprise Linux 6.X 8 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 28 GB VMDK on datastore2 (OS) 4 GB swap 24 GB / (ext4, room for VOM) Web Tier Red Hat Enterprise Linux 6.X 4 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 24 GB VMDK on datastore2 (OS) 4 GB swap 20 GB / (ext4) Application Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) NIC3 on vswitch3 (VCS HB1) NIC4 on vswitch4 (VCS HB2) 80 GB VMDK on datastore3 (CFS/FSS, Application data) 32 GB VMDK on datastore2 (OS) 8 GB swap 24 GB / (ext4, room for application binaries) Database Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) VM DirectPath Infiniband Port1 (VCS HB1) VM DirectPath Infiniband Port2 (VCS HB2) VM DirectPath SSD (CFS/FSS, Oracle data) 8
32 GB VMDK on datastore2 (OS) 8 GB swap 24 GB / (ext4, room for Oracle binaries) Recovery Site (Ashburn, VA): ESXi 5.1 build: 64 GB RAM 2 x Intel Xeon E5-2620 processors (6 core processors, with Hyperthreading capabilities, so VMware sees it as 24 logical processors) 300 GB SAS Disk #1: 10 GB ESXi partition, remainder VMFS datastore1 300 GB SAS Disk #2: VMFS datastore2 300 GB SAS Disk #3: VMFS datastore3 300 GB SAS Disk #3: VMFS datastore3 ESXi Host #1 Guest Virtual Machines: vcenter 5.1 Windows 2008R2 8 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 50 GB VMDK on datastore2 (OS) Symantec HA Console Windows 2008R2 4 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 50 GB VMDK on datastore2 (OS) Active Directory/DNS Windows 2008R2 2 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 40 GB VMDK on datastore2 (OS) Veritas Operations Manager (VOM) Red Hat Enterprise Linux 6.X 8 GB RAM 9
2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 28 GB VMDK on datastore2 (OS) 4 GB swap 24 GB / (ext4, room for VOM) Web Tier Red Hat Enterprise Linux 6.X 4 GB RAM 2 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) 24 GB VMDK on datastore3 (OS) 4 GB swap 20 GB / (ext4) Application Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) NIC3 on vswitch3 (VCS HB1) NIC4 on vswitch4 (VCS HB2) 80 GB VMDK on datastore3 (VxVM/VxFS, Application data) 32 GB VMDK on datastore3 (OS) 8 GB swap 24 GB / (ext4, room for application binaries) Database Tier Red Hat Enterprise Linux 6.X 16 GB RAM 4 vcpu NIC1 on vswitch1 (User) NIC2 on vswitch2 (Management) NIC3 on vswitch3 (VCS HB1) NIC4 on vswitch4 (VCS HB2) 300 GB on datastore4 (VxVM/VxFS, Oracle Data) 32 GB VMDK on datastore3 (OS) 8 GB swap 24 GB / (ext4, room for Oracle binaries) 10
Appendix 2 - Primary Database Cluster Configuration 11
12
13 Datacane - When the bits hit the fan!
14
15 Datacane - When the bits hit the fan!
16
Appendix 3 - Primary Application Cluster Configuration 17
18
19 Datacane - When the bits hit the fan!
20
21 Datacane - When the bits hit the fan!
22
23 Datacane - When the bits hit the fan!
24
Appendix 4 - First Primary Web ApplicationHA Configuration 25
26
Appendix 5 - Second Primary Web ApplicationHA Configuration 27
28
Appendix 6 - Recovery Site Single Node DB Cluster Configuration 29
30
31 Datacane - When the bits hit the fan!
32
Appendix 7 - Recovery Site Single Node App Cluster Configuration 33
34
35 Datacane - When the bits hit the fan!
36
37 Datacane - When the bits hit the fan!
38
39 Datacane - When the bits hit the fan!
Appendix 8 - Recovery Site Web Server ApplicationHA Configuration 40
41 Datacane - When the bits hit the fan!
Appendix 9 - Primary Database Server Volume Manager (VxVM) Configuration 42
43 Datacane - When the bits hit the fan!
44
45 Datacane - When the bits hit the fan!
Appendix 10 - Recovery Site Database Server Volume Manager (VxVM) Configuration 46
47 Datacane - When the bits hit the fan!
48
Appendix 11 - Primary Application Server Volume Manager (VxVM) Configuration 49
50
Appendix 12 - Flexible Storage Sharing Configuration The screen capture below shows the storage configuration as viewed by the first database node. As mentioned earlier, Flexible Storage Sharing (FSS) was employed to share each node s direct-attached storage (DAS) with the other node in the cluster. In this case, each node has two Intel solid state disk (SSD) devices which it has exported to the other node. So from system ia-db1, we see that in addition to its own Intel SSD devices named sdc and sdd, ia-db1 also has as remote disks the Intel SSD devices exported to the cluster from ia-db2. When volumes are created in an FSS configuration they are created on a local disk, and mirrored to a remote disk. In this way the volume is still available to surviving nodes in the event a node is lost. File systems created on these volumes are concurrently mounted on all nodes in the cluster. Symantec Cluster File System supports up to eight nodes when FSS is employed. Figure 6: Output from vxdisk command showing FSS-shared disks 51
Appendix 13 - Stills From The Datacane Video Figure 7: The primary data center, before the storm Figure 8: Hurricane in full swing, with winds increasing by the minute 52
Figure 9: Chad Bersche (seated, left) and John Cronin (seated, right), both Symantec Product Managers, monitor the progress of the failover of the Norton e- store applications as Amol Shah looks on Figure 10: With eight seconds remaining on the clock, the storm surge arrives, posing an imminent threat to the primary data center 53
Figure 11: Out of time, the flood waters enter the data center Figure 12: This really can t be good for data center equipment 54
Figure 13: Jason Buffington (left), Sr. Analyst with Enterprise Strategy Group, and Ryan Jancaitis, Symantec Product Manager, monitor the status of the Norton e-store service from Symantec corporate headquarters in Mountain View, California and confirm that the service is back online and running at 55
About Symantec Symantec Corporation (NASDAQ: SYMC) is an information protection expert that helps people, businesses, and governments seeking the freedom to unlock the opportunities technology brings anytime, anywhere. Founded in April 1982, Symantec, a Fortune 500 company operating one of the largest global data-intelligence networks, has provided leading security, backup, and availability solutions for where vital information is stored, accessed, and shared. The company s more than 20,000 employees reside in more than 50 countries. Ninety-nine percent of Fortune 500 companies are Symantec customers. In fiscal 2013, it recorded revenues of $6.9 billion. To learn more go to www.symantec.com or connect with Symantec at: go.symantec.com/socialmedia. For specific country offices and contact numbers, please visit our website. Symantec World Headquarters 350 Ellis St. Mountain View, CA 94043 USA +1 (650) 527 8000 1 (800) 721 3934 www.symantec.com Copyright 2014 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo, and the Checkmark Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. 5/2014 21330246