How To Recover From A Disaster With An Org Powerpoint Server With An Omerc Recoverpoint

Transcription

1 Disaster Recovery of Oracle Fusion Middleware and Oracle Database Server with EMC RecoverPoint Applied Technology Abstract Oracle database and application administrators face many challenges to managing the application and storage resources necessary for Oracle operations. This white paper outlines how EMC RecoverPoint provides costeffective local and remote replication of Oracle Fusion Middleware and Oracle Database Server as part of a disaster recovery solution. August 2009

2 Copyright 2009 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number h6450 Applied Technology 2

3 Table of Contents Executive summary...5 Background... 5 Introduction...5 Audience... 6 RecoverPoint...6 Advantages of RecoverPoint... 6 Local and remote recovery... 6 Replication modes... 7 Oracle database protection with RecoverPoint... 7 Federated environments and consistency groups... 8 Use case objectives...10 Oracle/EMC environment block diagram...10 RecoverPoint validation architecture...12 CLARiiON array-side preparations...14 System configuration and storage...15 Hardware configuration Software configuration Oracle Fusion Middleware, web tier, and Database volume layout RecoverPoint CRR consistency groups RecoverPoint group sets and parallel bookmarks OCFS2 configuration...19 High availability for persistent stores...19 Use high-availability storage for state data Network configuration...20 Planning for disasters and planned downtime...20 Initiate the replication process RecoverPoint software installation and configuration Starting replication Switchover procedures Switchback procedures Failover procedures Failback procedures General recommendations...28 Setting snapshots or manual bookmarks based on requirements Periodic DR testing Event notification General I/O and sizing Conclusion...29 References and resources...30 Oracle Applied Technology 3

4 EMC Appendix...31 Oracle DR terminology RecoverPoint terminology RecoverPoint write splitters RecoverPoint image access modes Virtual access (instant) Virtual access (instant) with Roll image in background Logged access (physical) Disable Image Access...35 Applied Technology 4

5 Executive summary Oracle s family of middleware products is comprehensive, standards-based application infrastructure software from the leading Java application server to SOA and Enterprise 2.0 portals. Pre-integration with Oracle Applications, Database, and Enterprise Manager speeds implementation and lowers costs. EMC RecoverPoint is an important advancement in the area of data replication. RecoverPoint provides local and remote replication for heterogeneous servers and storage and enables multiple applications to have replication consistency with fine-grained control over local and remote recovery. This white paper describes an application topology supporting core services provided by Oracle Fusion Middleware and Oracle Database Server. The data replication and disaster recovery products represented can address your specific application topology as it expands beyond the core represented here. This disaster recovery validation test incorporates the use of Oracle Application Server SOA Suite and Oracle Database Server 10gR2. Oracle SOA Suite is a comprehensive, hot-pluggable software suite for the building, deployment, and management of a service-oriented architecture. This includes service-oriented application development, service-oriented applications and IT systems integration, and service-oriented management of business processes. It plugs in to a heterogeneous IT infrastructure and enables enterprises to adopt SOA incrementally. These applications rely on hundreds of gigabytes or even terabytes of data and they share one common factor. They need a well-designed recovery plan in case of disaster. Oracle and EMC deliver key solutions that address the protection of mission-critical applications. Enterprise deployments need protection from unforeseen disasters and natural calamities. One protection solution involves setting up a disaster recovery (DR) site at a geographically different location from the production site. The DR site may have equal or fewer services and resources compared to the production site. Application data, metadata, configuration data, and security data are replicated to the DR site on a periodic basis. The DR site is normally in a passive mode; it is started when the production site is not available. This deployment model is sometimes referred to as an active/passive model. This model is normally adopted when the two sites are connected over a WAN and network latency does not allow clustering across the two sites. Background This white paper describes the setup and testing environment deployed to validate EMC RecoverPoint 3.1 with both Oracle Fusion Middleware and an Oracle database. This test configuration validates that the entire environment can be restored at a secondary site (DR site) in the event of a major production site failure. This test validates third-party replication using EMC RecoverPoint and Oracle Database Server as a joint solution that protects both database and non-database artifacts. The white paper Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint outlines an alternative solution approach where Oracle Data Guard is used to protect Oracle database files. Introduction This white paper is a follow-on paper to Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint. It addresses and explains the benefits of using EMC RecoverPoint local and remote replication to provide operational and disaster recovery for the Oracle Fusion Middleware and database environments. RecoverPoint provides application-consistent recovery points that can be utilized in response to a number of possible scenarios, enhancing the native availability of Oracle environments. Applied Technology 5

6 Audience The intended audience for this paper includes both storage administrators as well as database administrators seeking to understand best practices for failover and recovery of both Oracle Database Server and Oracle Fusion Middleware environments. The reader will establish an understanding of the importance of using an integrated solution to ensure effective recovery of all necessary data files to restart not only Oracle databases, but the applications they support. Detailed best practices are provided based upon EMC and Oracle joint testing, helping provide IT organizations with guidelines to develop a complete disaster recovery solution specific to their business. RecoverPoint RecoverPoint is proven technology for high-availability Oracle environments with both local and remote protection across SAN storage with complete protection against many possible disaster scenarios. This type of environment provides resiliency against failures within the data center infrastructure. It can help improve recovery from a regional disaster, all with the added benefit of immediate and instantaneous application recovery. Oracle products are inherently highly available and provide enterprise-class reliability without compromising security, performance, or scalability. To enhance the built-in availability features for Oracle, consider the following requirements for a data protection solution: Protection from infrastructure failure (such as a storage array or SAN switch) Protection from local or regional disaster Protection from data corruption Many companies are deploying continuous data protection (CDP) as a way to meet their recovery time objectives (RTO) and recovery point objectives (RPO). A true CDP implementation ensures that all changes to an application s data are tracked and retained consistently. In effect, CDP creates an electronic journal of application data snapshots, one for every instant in time that a modification occurs. Advantages of RecoverPoint RecoverPoint CDP preserves a record of the write transactions that take place within its environment, providing crash or application consistent recovery points. For local replication, RecoverPoint captures every write with I/O splitter technology (see the RecoverPoint write splitters section on page 33) and preserves them in a local journal; for remote replication, transactions are grouped based on user-specified policies, with significant write changes preserved in a journal at the DR site. This preservation of writes ensures that if data is lost or corrupted, such as from a server failure, virus, Trojan horse, software errors, or end-user errors, it is always possible to recover a clean copy of the affected data. Another advantage for RecoverPoint is that this data recovery can be performed at either the local or remote locations. These recovery points can be immediately accessed and mounted back to production environments in seconds much less time than is the case with disk-based snapshots, tape backups, or archives. Local and remote recovery EMC RecoverPoint is a comprehensive data-protection solution providing concurrent local and remote (CLR) data protection. This integrates both CRR (remote) and CDP (local) replication, allowing users to recover applications to any point in time. The integration of local (CDP) and remote (CRR) replication protects data against catastrophic events that can bring entire data centers to a standstill. RecoverPoint delivers superior data protection by allowing both local and remote replication with no application performance degradation. As a result, organizations can deploy geographically dispersed data centers for maximum protection from local or regional failure or disaster. Applied Technology 6

7 RecoverPoint CDP, CRR and CLR use the same write-splitting methods, journaling technologies, and appliance platform. They both make use of consistency groups, which let the user define sets of volumes for which the inter-write order must be retained during replication and recovery. This ensures that data at any point in time will be fully self-consistent. With RecoverPoint CDP, instead of compressing the data and sending over an IP network to a remote volume, it writes the data to a local journal and then to a local volume. As there is no IP network involved, and hence no latency concern, RecoverPoint CDP can synchronously track every write in the local journal and distribute the write to the target volume all without impacting application-server performance. RecoverPoint CRR transfers significant writes, based on an application s RPO/RTO, to a remote site where they are saved in a history journal. Once the appliance receives the write, it will bundle this write up with others into a package. Redundant blocks are eliminated from the package, and the remaining writes are sequenced and stored with their corresponding timestamp and bookmark information. The package is then compressed, and an MD-5 checksum is generated for the package. The package is then scheduled for delivery across the IP network to the remote appliance. Once received the remote appliance verifies the checksum to ensure the package was not corrupted in the transmission. The data is then uncompressed and written to the journal volume. Once the data has been written to the journal volume, it is distributed to the remote volumes, ensuring that write-order sequence is preserved. RecoverPoint CRR enables the user to perform backups at the remote site, eliminating the need to take the production file systems offline. With RecoverPoint data recovery can be performed locally and/or remotely by rewinding the target volumes back to a selected point in time by using earlier versions of data saved in the journal. RecoverPoint CRR provided protection for Oracle Fusion Middleware data. It validated that an entire environment could be restored at a secondary or disaster recovery site, in the event of a major production site failure. Replication modes When transferring data from the local site to the remote site, the RecoverPoint system automatically switches between replication modes on the fly to ensure that it is always using the method that best fits the current load conditions and the replication policy. The RecoverPoint system automatically switches between the following modes, according to load conditions: Continuous synchronous mode Continuous asynchronous mode Snapshot mode RecoverPoint automatically uses the replication mode that is most effective for the current conditions, including the application load, throughput capacity, and replication policy. Regardless of the replication mode, RecoverPoint is unique in its ability to guarantee a consistent replica at the target side under all circumstances, and in its ability to retain write order fidelity in multi-host heterogeneous SAN environments. For more information regarding replication modes, please refer to the EMC RecoverPoint Release 3.1 Administrator s Guide. Oracle database protection with RecoverPoint EMC RecoverPoint supports both single instance and Oracle Real Application Clusters (RAC) for local and remote replication of the Oracle RAC SAN-attached volumes. The Oracle LUNs are grouped into a single consistency group, and replication sets are created to map the production LUNs to the remote and/or local copy LUNs. RecoverPoint then processes the consistency group based on the type of recovery required. The following are among the options Oracle provides for protecting its databases, which can be integrated with RecoverPoint replication. Applied Technology 7

8 Application-consistent recovery from a shutdown (also known as cold backup) The user creates a consistency group that represents the Oracle application. The consistency group contains all volumes for the application, including data files, online redo log files, configuration files, and optionally control files. This method produces a copy from which you can restore the database with 100 percent reliability. Because normal operations must be halted while the cold backup is being created, this method is not appropriate for systems that must operate on a 24 x 7 basis. In addition, any changes protected by RecoverPoint that are made before or after the cold backup will not be available as an application-consistent recovery point, but they can be recovered as a crashconsistent recovery point (see below). When the database is shut down, the user will create a RecoverPoint bookmark for the specific consistency group, identifying the image as a cold backup image. This bookmark can be used to identify a point-in-time recovery image that represents a fully restorable and restartable Oracle database image. Crash-consistent recovery during operations This process enables the creation of crash-consistent images without requiring system shutdown. This process is performed by default by RecoverPoint for all applications as part of the RecoverPoint writesplitting operations. As a write is issued to the production volumes, RecoverPoint splitters intercept it and send a copy of the write to the RecoverPoint appliance for further processing. These captured writes represent the on-disk consistent data, which is the same data that remains on external storage even when an application crashes. When Oracle is restarted from a server crash, an instance must first open the database and then execute recovery operations. The instance automatically uses the redo log to recover the committed data in the database buffers that was lost when the instance failed. Oracle also undoes any transactions that were in progress on the failed instance when it crashed, then clears any locks held by the crashed instance after recovery is complete. When Oracle is restarted from a RecoverPoint crash-consistent image, it will perform the same recovery procedures. Application-consistent recovery during operation (also known as hot or fuzzy backup) This process enables the creation of application-consistent images without requiring system shutdown. It is required that all data files belonging to the relevant tablespaces, archive log files, and control files are flushed from the server s in-memory buffers to disk. To ensure that Oracle can recover from these images, Oracle must write additional information to the log file; information not required when crashconsistent images are sufficient. This feature in RecoverPoint requires that the user script several commands both to the Oracle Server and to the RecoverPoint appliance. The procedure entails placing the appropriate tablespace or database into backup mode (for example, ALTER TABLESPACE BEGIN BACKUP or ALTER DATABASE BEGIN BACKUP). When in hot backup mode, when a database block is modified, the entire block is written to the online redo logs. In normal operations, only the changed bytes are written. Also the data file headers are not updated with the SCN when a checkpoint is performed. Once Oracle backup mode is set, the script creates a RecoverPoint bookmark for the specific consistency group to identify the image as an application-consistent image. Archived redo logs can be used against this image. Federated environments and consistency groups Federated environments are related applications that span multiple servers and storage arrays at the same site. Each application has its own RPO and RTO policies that govern the protection and recoverability of the application s data. In RecoverPoint terminology, each application becomes a RecoverPoint consistency group with its own policy, journal, and replication set. For a successful recovery of the federated environment the customer must ensure that the individual consistency groups have a common recovery point across all of the applications. To get a common recovery point, the user creates a group set that defines the RPO across all of the RecoverPoint consistency groups that make up the federated environment. This is done though the use of a common bookmark in each journal for the consistency groups that are part of the group set. Applied Technology 8

9 When the user recovers each of the consistency groups that make up the federated environment and selects the common bookmark, the user will ensure that all of the application s data is recovered to the exact same point in time. This enables the data to be used for such advanced features as building a testing and development environment, creating a federated backup image, data mining, and so forth. Use of federated consistency groups provides the following: Allows application recovery to be tiered by service level Multiple volumes per group Mixed recovery point objectives within the same infrastructure Provides independent replication controls Recover by group, either locally or remotely Start/stop by group Enables grouping of optimization Importance Resource usage Recovery point and recovery time objectives Each tier can have different service level agreements Consistency groups per tier Operational recovery of tier Enforces consistency across tiers Federated environments Recover to a known point for all applications Disaster recovery for tier or application Spans operating systems, applications, storage, and servers Enables advanced functions Full environment cloning Application upgrade testing Data mining Consistent production rebuild The federated environment topology consists of three separate tiers: the web tier, the Oracle Fusion Middleware SOA Suite application tier, and the database tier. For successful recovery, consistency must be maintained between the middleware application tier and the database tier. Applied Technology 9

10 Use case objectives Objective Validate the test system architecture deploying key Oracle and EMC technologies. Details The archtecture consists of SAN-based storage with multiple Linux servers deployed in a production / DR site mode. Utilize EMC RecoverPoint to protect all tiers, that is,web, Oracle Fusion Middleware, and database. Ensure a very small to zero RTO and a very small RPO. Provide no single point of failure. Support the replication of WebLogic configuration and transaction logs (TLogs) Describe how RecoverPoint is set up and administered to protect the Oracle WebLogic application server, configuration information, and the WebLogic persistent stores (transaction logs), the SOA Suite binaries and configuration, and the Oracle database. Detail how RecoverPoint CLR is used to create simultaneous local/remote replicas and bookmarks. Accomplish with clustered WebLogic Application servers, EMC storage as well as highly available replication appliance (RPA) configurations. Achieve through EMC RecoverPoint CLR for local and remote replication. Set up procedures to demonstrate planned switchover and switchback and unplanned failover and failback. Show these procedures using EMC RecoverPoint. Demonstrate ease of use in WebLogic and Oracle database environment failover and failback procedures. Run the Oracle validation test suite, verifying failover data integrity. Document these procedures with easy-to-follow failover and failback instructions. Execute test program and procedures that validate RecoverPoint as a DR solution. This paper describes a test environment that is representative of DR protection of a production environment. The actual validation tests are not addressed here. Oracle/EMC environment block diagram Figure 1 depicts the major components and relationships between the servers, and applications on the primary/production and disaster recovery sites. The configuration consists of a single web host and two application servers with Oracle Fusion Middleware. They are clustered using Oracle WebLogic Server clustering. EMC RecoverPoint protects all objects including the Oracle Home binaries, Oracle Fusion Middleware, and the single instance database. Applied Technology 10

11 Clients Primary Site Standby Site Load Balancer Load Balancer Web Host APACHE SERVER Web Host APACHE SERVER Web Host APACHE SERVER Web Host APACHE SERVER App Host # 1 App Host # 2 App Host # 1 App Host # 2 RecoverPoint Manager Admin Server RecoverPoint Manager Admin Server Cluster Cluster RecoverPoint Appliance BPEL+ ESB + OWSM BPEL+ ESB + OWSM RecoverPoint Appliance BPEL+ ESB + OWSM BPEL+ ESB + OWSM RecoverPoint (RP) CRR Database Fusion Middleware & Logs Data Guard Replication Database Figure 1. Topology diagram of an SOA Suite Application and RecoverPoint use case environment Applied Technology 11

12 RecoverPoint validation architecture The RecoverPoint architecture for the middleware disaster recovery consists of four RecoverPoint appliances (RPAs) attached to both primary and secondary storage through Fibre Channel with two paths to either appliance. The connections are managed through a Brocade DCX and 7600 switch. They have an IP connection on a private subnet through a gige dlink switch to simulate a WAN link; this is in addition to the management IP connection that runs through a Cisco Catalyst switch. Both storage arrays, the EMC CLARiiON CX4-480 and CX4-960, have the integrated RecoverPoint write-splitting technology. Each storage array has a storage group created specifically to mask the LUNs to the RecoverPoint appliances as well as the middleware hosts. Figure 2. RecoverPoint use case architecture All the hosts are zoned through the Brocade switches to the CX-480 and CX-960, using the switch explorer GUI. The RecoverPoint appliances were also zoned over to the storage in the same manner. For each LUN masked to the RecoverPoint appliances, a replication set is created linking the source LUN to a target LUN. These replication sets are all placed within the middleware consistency group. The CX splitter is integrated with each storage processor (SP) of the CLARiiON array and this will send one copy of the write to the storage array and the other to the RecoverPoint appliance. The following types of storage volumes are required for RecoverPoint configuration: Repository volume: This volume holds the configuration and marking information during replication. At least one repository volume is required per site and is accessible from all the RPAs at the site. Journal volume: This volume is used to store all the modifications. The application-specific bookmarks and timestamp details are written to the journal volume. The size of the journal depends on the write activity to the protected LUN(s) and the RPO required for the data. Best practices information on the sizing and configuration of journal volumes is available in the EMC RecoverPoint Installation Guide. Applied Technology 12

13 Replication set: The association created between the source volume and the local and/or remote target volumes is called the replication set. A consistency group contains one or more replication sets. Consistency group: The logical grouping of replication sets identified for replication is called a consistency group. Consistency groups ensure that the updates to the associated volumes are always consistent with write-order preserved and that they can be used to restore the database at any point of time. The following shows the Middleware consistency group and its assigned replication sets from the RecoverPoint Management Application GUI. The names of the volumes in the Middleware and DR columns are derived from the friendly names of the LUNs returned by the CLARiiON array during the SCSI discovery operations. The number in parenthesis is the CX LUN ID number. Figure 3. Consistency group and replication set definitions Applied Technology 13

14 The following RecoverPoint Management Application GUI screenshot shows the Middleware consistency group storage volumes replicated by RecoverPoint (defined by replication sets). Figure 4. Storage volumes in replication sets CLARiiON array-side preparations The integrated RecoverPoint write-splitting technology is enabled on each storage processor (SP) of the CLARiiON array. The integrated write-splitter will intercept every write to a protected LUN and will send a copy of the write to the RecoverPoint appliance and send the original to the protected LUN. This setup has a CX4-480 as primary storage and a CX4 960 at the DR site. RecoverPoint 3.1 is installed and configured to utilize the integrated RecoverPoint write-splitting technology in the CX4 arrays. Applied Technology 14

15 System configuration and storage Hardware configuration Table 1. Server types used Dell PowerEdge 2950 Dell PowerEdge Quad-Core CPUs, 4 Quad-Core CPUs, 2.50 GHz, 16 GB RAM, 2.93 GHz, 32 GB RAM, Brocade 8 GB HBAs Brocade 8 GB HBAs Table 2. Fibre Channel switch types Primary DR Brocade DCX Brocade 7600 Table 3. Storage types used Primary CLARiiON CX4-480 FLARE 28 DR CLARiiON CX4-960 FLARE 28 Table 4. Server functions and types Server function Server type Production Site Apache Web Server Dell 2950 Production Site Oracle WebLogic Server Cluster 1 Production Site Oracle WebLogic Server Cluster 2 Dell 2950 Dell 2950 Production Site Oracle Database Server Dell 900 DR Site Apache Web Server Dell 2950 DR Site Oracle WebLogic Server Cluster 1 Dell 2950 DR Site Oracle WebLogic Server Cluster 2 Dell 2950 DR Site Oracle Database Server Dell 900 Applied Technology 15

16 Software configuration Table 5. Software versions and configuration Software Version Configuration OEL Enterprise Linux Server (Carthage) Release 5.2 EMC PowerPath 5.1 SP 2 EMC RecoverPoint CLR 3.1 x86_64 Oracle Database 10g Release Linux X86_64 Oracle Database Server Patch Set Patch Set Oracle Enterprise Manager 10g Grid Control Oracle Enterprise Manager 10g Grid Control Agent Linux X86_ Linux x86_64 Oracle WebLogic Server 9.2 MP3 Linux x86 Oracle Application Server SOA Suite 10g Release 3 Oracle Application Server Patch Set Oracle SOA Suite for WebLogic Server 9.2 MP3 Patch Oracle Patch Resolve Prereq Installation Issue Apache HTTP Server Linux x Patch Set Patch Patch Linux x86_64 Oracle JDeveloper 10g Release 3 Studio Version Windows XP Service Pack 2 Apache ANT Applied Technology 16

17 Oracle Fusion Middleware, web tier, and Database volume layout Table 6. Volume layout Volume name VolWeb VolAdmin VolWLS1 VolWLS2 VolData VolOrcl1 VolOrcl1 Oradata_1 Oradata_2 Oradata_3 Oradata_4 Tier Size Mounted on nodes Web Tier App Tier App Tier App Tier App Tier App Tier App Tier DB Tier DB Tier DB Tier DB Tier Mount point Notes/Comments 20G webhost /u01/app/oracle Volume for Apache Install 20G apphost1 /u01/app/oracle/wls/soadomain/admin Volume for Admin server Instances 20G apphost1 /u01/app/oracle/wls/soadomain/mng1 Volume for Managed Server Instance 20G apphost2 /u01/app/oracle/wls/soadomain/mng2 Volume for Managed Server Instance 20G apphost1, apphost2 Journal RP 50G RP appliance Journal RP 50G RP appliance Journal RP 50G RP appliance Journal RP 50G RP appliance /u01/app/oracle/data Volume for TLogs and JMS Data 20G apphost1 /u01/app/oracle/product Volume for binaries, both Oracle and WebLogic 20G apphost2 /u01/app/oracle/product Volume for binaries, both Oracle and WebLogic 100G dbhost /u01/app/oracle Volume for Oracle binaries and Flash Recovery Area 100G dbhost /u01/oradata/orclsoa/cg1_dbf_undo Database Files and Undo 50G dbhost /u01/oradata/orclsoa/cg2_redo Database Online Redo Logs 100G dbhost /u01/oradata/orclsoa/cg3_arch_ctl Database Archive Logs and Controlfiles Raw Volume - Fusion Middleware Consistency Group Production Raw Volume - Fusion Middleware Consistency Group CRR Replica Raw Volume - Database Consistency Group, cg1_dbf_undo Production Raw Volume - Database Consistency Group, cg1_dbf_undo CRR Replica RecoverPoint metadata RecoverPoint metadata RecoverPoint metadata RecoverPoint metadata Journal RP 50G RP Raw Volume - Database Consistency RecoverPoint Applied Technology 17

18 Journal RP 50G RP appliance Journal RP 50G RP appliance Journal RP 50G RP appliance appliance Group, cg2_redo Production metadata Raw Volume - Database Consistency Group, cg2_redo CRR Replica Raw Volume - Database Consistency Group, cg3_arch_ctl Production Raw Volume - Database Consistency Group, cg3_arch_ctl CRR Replica RecoverPoint metadata RecoverPoint metadata RecoverPoint metadata Please note the VolData volume is mounted simultaneously on both application hosts. This volume contains the WebLogic persistent file-based store. File-based stores must be configured on shared storage. The OCFS2 clustered file system is configured for the shared storage. All server root volumes are installed on local disk. Although not used in this validation, RecoverPoint does support the replication of root volumes installed in the SAN (boot from SAN). RecoverPoint CRR consistency groups For this use case three additional consistency groups for the database tier were created. The Oracle Fusion Middleware tier consists of one consistency group as previously configured. The white paper Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint has further detail. The Oracle Fusion Middleware has two types of data that are being replicated. The first is the SOA Suite, WebLogic binaries, and configuration files. The second is the persistent file-based store used for the Java Message Services (JMS) and Transaction Logs (TLogs). The persistent file-based store is used to recover transactions. The RecoverPoint consistency groups used for the database tier will be provisioned for 1) data files and undo logs, 2) online redo logs, and 3) archived redo plus control files. This will provide for the various strategies of disaster recovery according to the business needs of the organization, that is, applicationconsistent recovery, crash-consistent recovery, and crash-and-application consistent recovery. The online redo logs change frequently. In a RecoverPoint configuration, placing the online redo logs in a separate, lower-priority consistency group and giving priority to the data and archive redo log replication sets (crash-consistent strategy) may improve replication performance and slightly decrease time between snapshots (point-in-time recovery objective). The actual impact for all disaster recovery strategies depends on the bandwidth and the percentage of writes being sent over the link to the replica site. Please refer to the Replicating Oracle with EMC RecoverPoint Technical Notes. Each consistency group has settings and policies to include configuration group name, preferred RPA, and reservation support; and policies, such as compression, bandwidth limits, and maximum lag, which govern the replication process. In the configuration described above, the Lag time will need to be defined. This is the maximum offset between writing data to production storage and writing it to the RPA or journal at the replication site. The EMC RecoverPoint Release 3.1 Administrator s Guide has further detail on the settings and policies. RecoverPoint group sets and parallel bookmarks RecoverPoint group sets are used throughout a federated environment. Federated environments are related applications that may span multiple servers and storage arrays. Each application has its own RPO and RTO policies that govern the protection and recoverability of the application s data. After creating each consistency group with its appropriate settings and policies, to ensure that the individual consistency groups have a common consistency point across all of the applications, create a group set. The group set allows you to automatically bookmark a set of consistency groups so that the bookmark represents the same recovery point of each consistency group in the group set. This allows you to define Applied Technology 18

19 consistent recovery points for consistency groups that are distributed across different RPAs. The automatic periodic bookmark consists of the name you specified for the group set and an automatically incremented number. Numbers start at zero, and are incremented up to 65535, then begin again at 0. The same bookmark name is used across all consistency groups. To apply automatic bookmarks, the sources must be at the same site (replicating in the same direction) and transfer must be enabled for each consistency group included in the group set. The Group Set Details dialog box in the RecoverPoint Manager Application allows you to create, edit, or remove group sets. When creating the group set, enter a name for the automatic bookmarks. Select the consistency groups to be in the group set, and specify the bookmarking frequency. This will enable the parallel bookmarks. It is recommended that the interval between automatic bookmarks not be less than 30 seconds. If you prefer to take a bookmark at a specific time other than the automated times, or choose to manually take the parallel bookmarks, this can also be done through the RecoverPoint Manager Application. In the Navigation pane, select consistency groups. In the component pane, select all the consistency groups to bookmark simultaneously. All selected consistency groups must be enabled and transfer must be active. Click the parallel bookmarks icon in the upper right corner of the component pane. When prompted, enter the name of the bookmark. Please do not use the name latest as it is a reserved word in RecoverPoint. The group set can be perceived as a single entity, but each consistency group functions as a separate unit within the group set. The user recovers each of the consistency groups that make up the federated environment and selects the common bookmark. This will ensure all of the application s data is recovered to the exact same point in time. This can be executed through the RecoverPoint Manager Application or using the command line interface (CLI). Maintaining the autonomy of the consistency group in a group set provides the flexibility for each application to maintain its own policies and settings to govern the protection and recoverability of the application s data. It enables the data to be used for such advanced features as building a testing and development environment, creating a federated backup image, and data mining. OCFS2 configuration High availability for persistent stores The WebLogic application servers are usually clustered for high availability. For the local site high availability of the SOA Suite Topology, a persistent file-based store is used for the Java Message Services (JMS) and Transaction Logs (TLogs). This file store needs to reside on shared disk that is accessible by all members of the cluster, that is, by apphost1 and apphost2. The persistent file-based store can be migrated along with its parent server as part of the whole server migration feature that provides both automatic and manual migration. There are two methods for configuring a synchronous write policy: the Cache-Flush and the Direct-Write policies. The Cache-Flush policy improves performance, but the downside is possibly losing sent messages or generating duplicate messages in the event of an operating system crash or hardware failure. This is due to the fact that transactions are complete as soon as the writes are cached in memory, instead of waiting for acknowledgement that the writes are written to disk. Use high-availability storage for state data The server migration process moves or migrates services. Some state information associated with the work in process at the time of failure is persisted to storage. To ensure high availability, it is critical that such state information remains available to the server instance and the services it hosts after migration. It should be stored in a shared storage system that is accessible to any potential machine to which a failed migratable server might be migrated. For highest reliability, use a shared storage array solution (like EMC CLARiiON) that is itself highly available and a SAN designed for high availability. Applied Technology 19

20 If independent local file systems resided on a shared LUN, there would be no means of cache synchronization, and the file systems would eventually corrupt each other. Therefore, the shared storage solution in a SAN as deployed in this white paper uses the Oracle Cluster File System (OCFS2) as the hostbased clustered or shared file system technology. OCFS2 is a symmetric shared disk cluster file system that allows each node to read and write both metadata and data directly to the SAN. A shared storage solution in a SAN as described in this paper uses a host-based clustered or shared filesystem technology. Oracle Cluster File System (OCFS2) is used here, but any shared filesystem technology can be used such as Red Hat's GFS, Veritas VxCFS, or IBM GCFS. Network configuration The EMC RecoverPoint system has been designed as a secure platform for CDP (local), CRR (remote), and CLR (local and remote replication). EMC has invested in ensuring security for all aspects of its RecoverPoint system, including the operating system, networking, and RecoverPoint software. Security settings are divided into the following categories: RecoverPoint appliance (RPA) operating system and networking Access control settings to limit access by end users or by external product components Log settings related to the logging of events Communication security settings related to security for the product network communications Data security settings available to ensure protection of the data handled by the product Secure serviceability is available to ensure control of service operations performed on the products by EMC or its service partners. Other security considerations list known issues, including false positive findings that appear when scanning the product for vulnerabilities. For further discussion and implementation detail, please refer to EMC RecoverPoint Release 3.1 and Service Pack Releases Security Configuration Guide. Planning for disasters and planned downtime Initiate the replication process RecoverPoint is an intelligent data protection and recovery solution that runs on out-of-band appliances attached to the SAN and the IP network. RecoverPoint uses your enterprise s existing network and storage systems. To prepare for installation, you should be familiar with how the RecoverPoint system integrates with your existing systems. In preparation for RecoverPoint installation: 1. The RPAs connect to the hosts and storage subsystems using a Fibre Channel SAN. Before installing RecoverPoint, these subsystems should be in place. 2. RPAs are linked to the WAN interface using an Ethernet/IP connection (eth0). 3. Before you install RPAs and define consistency groups, ensure that sufficient volumes are available on the SAN-attached storage at each site for use by RecoverPoint. As part of the Applied Technology 20