REMOTE SITE RECOVERY OF ORACLE ENTERPRISE DATA WAREHOUSE USING EMC DATA DOMAIN

Transcription

1 White Paper REMOTE SITE RECOVERY OF ORACLE ENTERPRISE DATA WAREHOUSE USING EMC DATA DOMAIN EMC SOLUTIONS GROUP Abstract This white paper describes how a 12 TB Oracle data warehouse was transported from a data center in Massachusetts, U.S, to Cork, Ireland. The infrastructure was re-created at the Cork site on EMC Symmetrix VMAX storage and the database was restored and recovered. EMC Data Domain Encryption was used to secure the data during transport. July 2011

2 Copyright 2011 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 5 Business case... 5 Solution overview... 5 Key results... 6 Introduction... 7 Purpose... 7 Scope... 7 Audience... 7 Terminology... 7 Technology overview... 8 Overview... 8 EMC Data Domain DD EMC Data Domain Encryption... 9 EMC Data Domain file system encryption lock... 9 EMC Symmetrix VMAX... 9 EMC Solutions Enabler... 9 EMC Symmetrix TimeFinder... 9 EMC PowerPath Oracle Database Solution overview Use case requirements The challenge Nondisruptive to production Distance and latency Database size The solution Project timeline Discovery primary site Overview Security Business and technical Primary data center configuration information Remote site infrastructure Architecture Hardware resources Software resources

4 Backup appliance Oracle database backup primary site Overview DD660 setup Network settings Encryption RMAN backup File system lock Data compression results Oracle server deployment and storage provisioning remote site Overview Deploying the remote site server Configuring the remote site storage Primary data warehouse configuration information Provisioning storage at the remote site Setting up the Oracle database environment Oracle database recovery remote site Overview DD660 setup Network settings Unlock file system RMAN restore Verifying the integrity of the database Data Domain consolidation remote site Overview Replication process Conclusion Summary Findings References White papers Product documentation Other documentation

5 Executive summary Business case EMC offers an extensive series of solutions for quick, efficient, and nondisruptive backup of large Oracle data warehouse environments. Increasingly, customers are looking to repurpose or reuse these backups to facilitate operations such as: Migration of Oracle data warehouses to new data center locations Seeding of Oracle data warehouse environments at remote disaster recovery locations Deployment of remote database copies for test and development purposes These scenarios present challenges in relation to determining the most appropriate mechanism for transferring the data from the primary site to the remote site, and developing validated procedures for re-creating the production environment and recovering the database at the remote site. The main considerations in relation to data transfer to the remote site include: Security and corporate governance compliance Distance and latency Database size Cost effectiveness For one-off migration scenarios, where the data warehouse is many terabytes in size and the distance between data centers is great, it is impractical to transfer the data across the WAN. Consequently, customers are looking for an offline transport solution that is economical and secure. Solution overview At its headquarters in Hopkinton, Massachusetts, EMC IT maintains a 12 TB Oracle data warehouse, deployed on an Oracle Real Application Clusters (RAC) 11g environment. To demonstrate the benefits and implementation of the solution documented in this white paper, this real-world production database was cloned and backed up, shipped by air from the primary data center in Hopkinton, Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restored and recovered for test and development use. The remote data center infrastructure consisted of a two-node Oracle RAC 11g cluster that accessed the Oracle database on an EMC Symmetrix VMAX storage system. This reflects the infrastructure at the primary data center. The production database in Hopkinton was cloned using EMC TimeFinder and then backed up to an EMC Data Domain DD660 appliance using Oracle Recovery Manager (RMAN). Data Domain Encryption and file system encryption locking were used to ensure security and corporate governance compliance during transport. At the remote site, the DD660 was added to the environment and the file system was unlocked. The database was then restored and recovered to a test system using Oracle RMAN. 5

6 After verifying that the database had been successfully restored, Data Domain Replicator was used to move the database image from the DD660 to a shared Data Domain DD880 appliance in the data center in Cork, so that the DD660 could be repurposed for another project. Key results The key benefits demonstrated by the solution include: Nondisruptive backup Creating a TimeFinder clone has no impact on production. Mounting the clone to a separate backup server provides a fully-usable, independent, point-in-time copy that can be backed up to the Data Domain appliance with no disruption to the production environment. Reduced backup footprint and shipping costs Data Domain deduplication reduces the footprint of the backup database significantly. In this solution, a 70 percent reduction in capacity requirements enabled use of the midrange DD660 appliance, which minimized shipping costs. Seamless Oracle RMAN integration Data Domain systems integrate seamlessly into Oracle architectures, supporting existing backup strategies, including Oracle RMAN. Data security and corporate governance compliance Data Domain Encryption ensures that all data is physically stored in an encrypted manner and cannot be accessed without first decrypting it. Data Domain file system encryption lock adds a further level of protection that enables shipping on commercial air transport while satisfying corporate governance compliance requirements. Validated and documented end-to-end procedures Database integrity testing at the end of the project demonstrated successful recovery of the database at the remote data center. This white paper documents the procedures for achieving these results, from the initial discovery phase necessary for designing the infrastructure for the remote site, to a complete restore and recovery of a working data warehouse. These procedures are applicable to any solution involving recovery of an Oracle database. 6

7 Introduction Purpose This white paper documents a case study that involved cloning and backing up a realworld 12 TB database, shipping it by air from the primary data center in Hopkinton, Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restoring and recovering the database for test and development use. Scope The white paper covers the following: An overview of the technologies involved in the solution The discovery process carried out at the production site to develop a build plan for the remote site Re-creation of the data warehouse at the remote site Backup and securing of the DD660 for transfer by commercial air transport Restore and recovery of the database at the remote site Validation of the recovered database Replication of the database from the DD660 to a DD880 appliance Audience This white paper is intended for Oracle, storage, server, and backup administrators who want to understand how to migrate an Oracle database to a remote site and the build requirements at the remote site to enable recovery of the database. It is assumed that the reader is familiar with the following EMC and Oracle products: EMC Symmetrix VMAX EMC Data Domain Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware Terminology Table 1 defines terms used in this white paper. Table 1. Term AES ASM DD DD OS RAC RMAN SISL Terminology Definition Advanced Encryption Standard Automatic Storage Management Data Domain Data Domain operating system Real Application Clusters Recovery Manager Stream-Informed Segment Layout architecture 7

8 Technology overview Overview The case study uses the following hardware and software components: EMC Data Domain DD660 EMC Data Domain Encryption EMC Data Domain file system encryption lock EMC Symmetrix VMAX EMC Symmetrix TimeFinder EMC PowerPath Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware Oracle RMAN EMC Data Domain DD660 For the solution, an EMC Data Domain DD660 appliance was used to back up the source database and transport it to the recovery site in Cork. Data Domain deduplication storage systems dramatically reduce the amount of disk storage needed to retain and protect enterprise data, requiring a storage footprint that is 10 to 30 times smaller, on average, than that required by the original data set. The DD660 is a midrange appliance that offers up to 2 TB/hour of aggregate inline deduplication throughput, and up to 700 GB/hour for a single stream. The model used in the case study provides 8 TB of storage capacity, with much higher data storage capacity enabled by deduplication. Like all Data Domain systems, the DD660 derives its performance advantages from the Data Domain Stream-Informed Segment Layout (SISL) scaling architecture. SISL identifies 99 percent of duplicate segments in RAM, inline, ensuring that only unique data is stored to disk. The storage of unique data is invisible to backup software, which sees the entire virtual file system. The integrity and recoverability of backup data are secured by the Data Domain Operating System (DD OS) and Data Invulnerability Architecture, which ensures that data is stored and recoverable with continuous write verification, fault-detection, and self-healing mechanisms. When data enters the Data Domain system, using any of the supported protocols (NFS, CIFS, VTL, OST, and Distributed Segmented Processing), the stream is segmented, fingerprinted, deduplicated (Global Compression ), grouped into multisegment compression regions, and then locally compressed. When confidentiality is required, deduplicated and compressed data can be encrypted inline and the file system can then be locked. Data Domain systems integrate seamlessly with Oracle RMAN, with no changes to scripts, backup processes, or system architecture. 8

9 EMC Data Domain Encryption Data Domain Encryption protects data if the Data Domain system is stolen or lost during transit, and eliminates accidental exposure during replacement of failed drives. DD Encryption is designed such that if an intruder circumvents other network security controls and gains access to encrypted data, the data is unreadable and unusable to that person without the proper cryptographic keys. The inline encryption process integrates seamlessly with the inline deduplication process, encrypting data before it is written to disk. Similar to the advantages of inline deduplication, inline encryption requires minimal resources to provide fast, reliable, and secure backup and recovery. EMC Data Domain file system encryption lock File system encryption lock is specifically designed to lock an encrypted Data Domain appliance and its external storage prior to transporting the appliance between locations. File system encryption lock requires an encryption passphrase. This is a humanreadable key that is used to generate a machine-usable AES256 encryption key that encrypts the Data Domain system encryption key. Locking the file system involves changing the passphrase. This changes the encryption of the Data Domain system key and removes the passphrase from the system. Locking the system allows a Data Domain system to be transported with the encryption keys on the system. The keys are not recoverable until the file system is unlocked using the passphrase (which is not stored on the appliance). EMC Symmetrix VMAX An EMC Symmetrix VMAX provides the storage platform at both the primary and remote data centers. Symmetrix VMAX storage arrays provide high performance and scalability for demanding enterprise storage environments. Built on the strategy of simple, intelligent, modular storage, the Symmetrix VMAX incorporates a highly scalable Virtual Matrix Architecture that enables it to grow seamlessly and cost-effectively from an entry-level configuration into the world s largest storage system. The VMAX supports Flash drives, Fibre Channel drives, and SATA drives within a single array, as well as an extensive range of RAID types. EMC Solutions Enabler EMC Solutions Enabler is the software that provides a host with the Symmetrix Command Line Interface (SYMCLI), which is a comprehensive command set for managing your Symmetrix storage environment. SYMCLI is used in single command line entries and scripts to monitor device configuration and status, and perform control operations on devices and data objects within a managed storage system. EMC Symmetrix TimeFinder EMC Symmetrix TimeFinder software provides powerful replication capabilities for creating local point-in-time copies that are independent of the hosts and operating systems, applications, and databases. These copies are immediately available for read and write access. For the solution, a TimeFinder copy of the Oracle database was used to offload backup overhead from the production environment to another server. 9

10 EMC PowerPath EMC PowerPath is server-resident software that enhances performance and application availability by supporting multiple I/O paths to logical devices and intelligently distributing I/O requests across all available paths. PowerPath also provides automatic failover in the event of a hardware failure by automatically detecting the path failure and redirecting I/O to another path. For the solution, PowerPath provides intelligent path management and automatic path failover at the remote site. Oracle Database The solution involves an Oracle Database 11g R1 data warehouse and uses the features of Oracle ASMLib, Oracle Database 11g, and Oracle Clusterware 11g such as RAC, Automatic Storage Management (ASM), and RMAN. 10

11 Solution overview Use case requirements This solution is based on a real-world use case and is essentially a case study. The requirements of the use case were to take a production database and transfer it to a lab environment for test and development purposes. The EMC IT Production DB resides in EMC IT corporate headquarters in Hopkinton, Massachusetts, as shown in Figure 1. The test and development lab is located in the EMC Solutions Center in Cork, Ireland. The database is approximately 12 TB. Figure 1. The challenge An economical and secure mechanism was required to transport the data, and the process also had to have zero impact to the EMC IT production system. The challenge Nondisruptive to production As the project involved a real-world, production system, it was critical that the backup procedure did not impact database operations. This factor dictated a nondisruptive backup solution. Distance and latency The distance between the data centers in Hopkinton and Cork is approximately 4,740 kilometers (2,946 miles). The minimum latency for the round-trip is 47.4 ms, not including other delays introduced by encryption or the network itself. In fact, the average measured ping time between the data centers is 90 ms. For example, a dedicated WAN link, with 47.4 ms round-trip latency, 1 Gb/s bandwidth, and a TCP window size of 65,536 bytes, would restrict the maximum throughput to 1.3 MB/s and take approximately 112 days to move 12 TB of data. As this was a one-off project, it was not envisaged that sufficient dedicated bandwidth would be made available on the corporate WAN for the data migration. These factors dictated an offline method for transporting the data. 11

12 Database size The size of the database approximately 12 TB was a significant challenge and another factor that made it impractical to transport the data across the WAN. Figure 2 indicates just how big 12 TB is in real terms. Figure 2. What 12 TB looks like The solution To facilitate transporting the data in an economical and secure fashion, a Data Domain DD660 appliance was chosen. The reasons for this are: Small form factor The DD660 used has a 2U form factor and weighs 30 kg (66 lb) perfectly suited for low shipping costs. Data compression The DD660 has 8 TB of data storage space. However, Data Domain systems store only unique data, using Global Compression to pool redundant data from each backup image. This enables much higher data storage rates than the raw capacity of the device. Figure 3 shows the difference in size between the original database, the database backed up by RMAN, and the deduplicated, encrypted database on the DD660. Figure 3. Original and deduplicated database sizes 12

13 Data integrity The Data Domain Data Invulnerability Architecture protects against data loss from hardware and software failures. Encryption of data at rest Data Domain Encryption protects data if the Data Domain system is stolen or lost during transit. Encrypted file system lock Data Domain file system encryption lock protects an encryption-enabled Data Domain system (and its external storage devices) while in transit. The encryption keys are not recoverable without a passphrase, which is not stored with the system. To ensure that the production environment was not impacted by the project, it was decided to create a TimeFinder clone of the database and mount this to a backup server. This would provide a fully-usable, independent, point-in-time copy of the production database that could be backed up to the DD660 nondisruptively. Figure 4 illustrates the solution designed for the project and outlines the procedures involved. Figure 4. The solution 13

14 Project timeline The initial planning and feasibility phase of this project took place over several weeks prior to the official kickoff of the project. After the official kickoff, the entire project took just 22 working days to complete, from commencing discovery to verification of the restored database at the remote site. Figure 5 shows the project schedule. Predictably, discovery was the lengthiest phase (2 weeks) and was iterative in nature. It progressed in parallel with, and was informed by, the build process at the remote site. The comprehensive information gathering and planning during this phase was key to ensuring successful completion of the project. The result was a complete and successful restore and recovery of the database at the remote site in only 28 hours. Figure 5. Project timeline 14

15 Discovery primary site Overview This section describes the discovery process carried out at the start of the project to determine the feasibility of the project and to gather infrastructure and configuration information from the production site. This information was used to determine the transport mechanism between the two data centers and to create a build plan for the remote site. This phase of the project involved the steps shown in Figure 6. Figure 6. Discovery process The stakeholders involved in the project were: Primary site Stakeholders: EMC IT; EMC IT Security and Compliance Location: EMC IT headquarters, Hopkinton, Massachusetts, U.S. Database: EMC IT Production DB Remote site Stakeholders: EMC Global Solutions (Oracle Engineering) Location: EMC Solutions Center, Cork, Ireland (CSC) Database: CSC test and development database Security The security of the data to be transferred between the data centers was of critical importance to the feasibility of the project. EMC IT and EMC IT Security considered data security issues in terms of: Corporate governance and compliance that is, ensuring data security both in transit and at the remote data center so as to meet legislative requirements and EMC and industry standards. Identifying any potential personal/sensitive data for masking/cleaning. Choosing a backup/copy mechanism that would be nondisruptive to the production database. 15

16 This feasibility phase concluded that a copy of the EMC IT Production DB would be supplied to EMC Global Solutions under the following conditions: The data would be stored in a secure EMC data center at the remote site. Access to the data would be limited to specified EMC personnel. No data would be made public or available to external parties. A Data Domain DD660 would be used to transport the data, with Data Domain Encryption and Data Domain file system encryption lock enabled. Business and technical To enable recovery and repurposing of the data warehouse, the infrastructure at the primary site would be re-created at the remote site. In preparation for this, the following business and technical issues were considered: Mapping and configuration of the underlying infrastructure servers, storage, and IP and storage networks Identifying hardware and software versions for the: Server operating system Oracle database Symmetrix VMAX storage system Data Domain system Identifying the groups and individuals within EMC IT who could supply information and carry out the necessary tasks in the production environment A project plan was then agreed to that would minimize the impact on EMC IT staff and would move the data to Cork with minimal delay. Primary data center configuration information To facilitate the project, EMC IT made several important documents available to the EMC Global Solutions team in Cork. These documents detailed the design and architecture of the EMC IT data warehouse. The Detailed Solutions Design document for the data warehouse was key to understanding the production environment. This document contains the following information: High-level architecture Hardware inventory and configuration information for the database and backup servers and cluster, including CPU and memory, internal drives, and HBA and network cards and ports List of required Shared Infrastructure Services, including: SAN, storage array, and IP networks private (Cluster Interconnect), data center, and backup Storage rationale and design, hardware and software, and configuration Security user and group access and authentication Data center hardware provisioning environment and cabling details 16

17 In addition, the following live information was supplied: A copy of the current symapi_db.bin file from the production Symmetrix array An extract of the Oracle data dictionary views including V$ASM_DISKGROUP and V$ASM_DISK A listing of the Oracle ASM disks A mapping of the PowerPath devices and paths to the Symmetrix LUNs Multiple sample Automatic Workload Repository (AWR) reports A listing of the latest operating system and Oracle Clusterware and database patch levels Discovery was an iterative process, and continued in parallel with, and in response to, the remote site build. Remote site infrastructure Architecture From the information gathered during the discovery phase, the infrastructure for the remote site was designed as shown in Figure 7. Figure 7. Remote site architecture 17

18 Hardware resources Table 2 details the hardware environment for the remote site. Table 2. Hardware resources Purpose Quantity Resources Storage array 1 Symmetrix VMAX with: 2 x VMAX 64 GB engines 8 Gb FC connectivity 450 GB 15k FC drives Oracle RAC database servers 2 Linux server with: Quad-core CPU 96 GB RAM FC switches 2 8 GB/s FC switches Ethernet switches 2 1 Gigabit Ethernet switches Software resources Table 3 details the software resources for the remote site. Table 3. Software resources Software Red Hat Enterprise Linux 5.5 Oracle Database 11g R1 ( ) EMC Enginuity 5875 Purpose Server OS for database and application servers Database and cluster software Symmetrix VMAX operating environment Backup appliance Table 4 details the configuration of the Data Domain appliance used for backing up and transporting the database. Table 4. Appliance Data Domain appliance configuration Details EMC Data Domain DD x 1 TB SATA drives 1 Gb IP connectivity DD OS

19 Oracle database backup primary site Overview The process for configuring the DD660 and backing up the production database at the primary data center involved the steps shown in Figure 8. When the process was complete, the appliance was secure and ready to ship to Cork that is, the data was encrypted and the file system locked. Local site backup Configure DD660 network settings & add to network Enable DD660 file system encryption Create clone copy of production database RMAN backup to DD660 Test database restore DD660 secure & ready to ship Data encrypted File system locked Lock DD660 file system Backup complete Figure 8. Oracle database backup process DD660 setup Network settings The DD660 used for this solution had two 1 Gb network interfaces. To improve network performance and resiliency, these were aggregated and used in parallel. Figure 9 shows the link aggregation setup. Figure 9. DD660 link aggregation setup To enable backup of the database, the appliance was then provided with an IP address in the local network. 19

20 Encryption Data Domain Encryption was enabled prior to enabling the file system on the DD660. Data was encrypted using the 128-bit Advanced Encryption Standard (AES) algorithm (aes_128_cbc), as shown in Figure 10. This option strikes a reasonable balance between security and performance for the use case. Figure 10. DD660 encryption RMAN backup On the production site, a fresh clone copy of the storage was created using TimeFinder. An Oracle RMAN backup of this copy was then taken, with the database mounted, but not opened, on a backup server. A recovery catalog was used to record RMAN metadata. With Data Domain Encryption enabled, the data being backed up to the DD660 was deduplicated and then encrypted inline, before it was written to disk. The following were backed up to the DD660: A full RMAN backup of the Oracle database The database control file A schema export of the RMAN recovery catalog The Oracle team on the production site then validated the backup by restoring the control file and database to the backup server. Table 5 shows the duration of the backup and restore operations. Table 5. Process Backup and restore times Database backup to DD660 Timings 28 hours and 38 minutes Database restore (for verification) 30 hours and 2 minutes File system lock The filesys encryption lock command was used to lock the Data Domain file system. This required a passphrase change, as shown in Figure 11, automatically re-encrypts the system encryption key, and removes the new passphrase from the DD system. The new passphrase was communicated separately to the remote site. 20

21 Figure 11. Data Domain file system encryption lock Data compression results Data Domain deduplication eliminated redundant data segments during the backup process. This allowed the Data Domain appliance to store and manage much more data than would be possible with a traditional storage device. Figure 12 shows a DD OS view that summarizes data storage and compression statistics for the use case and illustrates the value of Data Domain deduplication when transporting an Oracle database. Table 6 also summarizes these statistics. Figure 12. Use case compression summary The total physical space available on the appliance is 7.5 TB, the total uncompressed data written by RMAN to the appliance was 10.6 TB, and the space required to store the compressed data was 3.2 TB. This demonstrates a 69.7 percent reduction in the space required to store the database. Table 6. Use case compression summary Item Total physical space available Total data written to appliance (uncompressed) Data stored on appliance (compressed) Value 7,528.6 GiB (7.5 TB) 10,613.6 GiB (10.6 TB) 3,217.0 GiB (3.2 TB) Deduplication factor 3.3 Percent reduction storage space saving 69.7% 21

22 Figure 13 shows the allocated, used, and free space information for the source database. This closely matches the statistics gathered by the DD660 and shown in Figure 12. Figure 13. SQL output showing database allocated and free space information The database had 11.6 TB of storage assigned to its data files, not including temp files, redo logs, control files and archive logs. Of this, TB was allocated to tables and indexes within the data files, with 1,504 GB of free space available. 22

23 Oracle server deployment and storage provisioning remote site Overview This section describes preparation of the hardware and software infrastructure at the Solutions Center lab in Cork, as outlined in Figure 14. Figure 14. Remote site preparation process Deploying the remote site server To simplify the restore process, the Oracle Solutions Engineering team decided to restore the database as a single instance before migrating it to Oracle RAC. The database server was configured with Red Hat Enterprise Linux 5.5 and prepared for a standard Oracle ASM/Database install. Server configuration involved the following steps: 1. Install and configure EMC PowerPath. 2. Install and configure ASMLIB. Configuring the remote site storage Primary data warehouse configuration information During the discovery phase of the project, EMC IT provided the Oracle Solutions Engineering team in Cork with the configuration database (symapi_db.bin) for the IT data warehouse storage array. At the remote site, the configuration information for the storage array was extracted by querying symapi_db.bin in offline mode using standard Solutions Enabler display commands. The procedure involved the following steps: 1. Copy symapi_db.bin to a host with Solutions Enabler installed. 23

24 2. Use the following Solutions Enabler commands to set up Solutions Enabler to query the database file: export SYMCLI_OFFLINE=1 export SYMCLI_DB_FILE= /DWproject/symapi_db.bin export SYMCLI_SID=XXX 3. Use the following command to extract the configuration details for all devices in the database storage group (GDWORAPRD), as shown in Figure 15: symsg show SG_GDWORAPRD Figure 15. Extracting device configuration information from a storage group With this information it was possible to understand how the production storage was provisioned. In conjunction with the other information sources supplied by EMC IT during the discovery phase, the Cork team was able to determine how much storage was required on the target Symmetrix VMAX, the Oracle ASM disk groups required, and the number of LUNs in each disk group. Provisioning storage at the remote site Provisioning storage at the remote site involved the following steps: 1. Create the devices on the target Symmetrix VMAX array, as listed in Table 7. Table 7. Storage assigned to Symmetrix VMAX array at remote site ASM disk group Number of devices Device size (MB) GDWP_TEMP 32 32,263 GDWP_ARCH 16 64,528 GDWP_REDO ,066 GDWP_REDO ,066 GDWP_T0_DATA ,056 GDWP_T1_DATA ,056 GDWP_T5_DATA ,056 Total Devices Configure each device with a device identifier label that corresponds to the name of the ASM disk group to which the device will belong. This label can be read from the host using the Inquiry utility, which makes device identification from the host much easier. 24

25 Figure 16 shows these labels being created with Solutions Enabler. Figure 16. Setting device identifiers with Solutions Enabler 3. Create a masking view for the devices so that they are visible to the RAC cluster. At this point, the storage is ready to be incorporated into the Oracle configuration at the remote site. Setting up the Oracle database environment Setting up the Oracle database environment at the remote site involved the following steps: 1. Partition and assign the PowerPath devices for use with Oracle ASM. Figure 17 shows the mapping of the Oracle ASM disks to the Symmetrix assigned device_name. Figure 17. Oracle ASM disk mapping to Symmetrix devices 2. Install Oracle Clusterware and the database binaries. 25

26 3. Configure the ASM instance and create the ASM disk groups listed in Table 8. Table 8. Disk group ASM disk groups and sizing Allocation unit (MB) Size (GB) GDWP_ARCH 1 1,008 GDWP_REDO GDWP_REDO GDWP_T0_DATA ,016 GDWP_T1_DATA ,081 GDWP_T5_DATA ,041 GDWP_TEMP 1 1,008 For example, Figure 18 lists the 16 Oracle ASM disks assigned to disk group GDWP_ARCH. Figure 18. Oracle ASM disks assigned to disk group GDWP_ARCH 26

27 Oracle database recovery remote site Overview This section describes the restore and recovery of the Oracle database at the remote site, as outlined in Figure 19. Figure 19. Database restore and recovery process DD660 setup Network settings When the appliance arrived at the remote data center, the IP address had to be changed to enable connectivity to the remote network. Link aggregation was already in place, so this was simply a case of changing the IP address of the virtual interface, as shown in Figure

28 Figure 20. Reset the IP address Unlock file system The Data Domain file system was locked during transit and could not be enabled without the passphrase, as indicated in Figure 21. The passphrase was sent to Cork separately from the DD660 appliance. Figure 21. Data Domain file system enable refused passphrase required The filesys encryption unlock command allows the user to enter the passphrase to unlock the file system, as shown in Figure 22. Figure 22. Unlocking and enabling the Data Domain file system 28

29 RMAN restore With the Data Domain file system unlocked, the database was then restored from the DD660 using RMAN. This process involved the following steps: 1. Mount the DD660 on the /backup directory, as shown in Figure 23. Figure 23. Mounting the DD660 on the remote site server 2. Create the Oracle database and build the RMAN recovery catalog from the export file held on the DD660. Figure 24 shows a summary listing of the database and control file backups from the catalog. Figure 24. Summary of database and control file backups 3. Use RMAN to connect to the target database and the recovery catalog. 4. Start up the Oracle instance without mounting the database (nomount mode), as shown in Figure

30 Figure 25. Starting up the Oracle instance in nomount mode 5. Restore the backup control file to the control_file location specified in the SPFILE, as shown in Figure 26. Figure 26. Restoring the backup database control file 6. Allocate the channels and restore the database, as shown in Figure

31 Figure 27. Database restore 7. Mount and open the database, as shown in Figure 28. Figure 28. Mounting and opening the database In this use case, further recovery was not required as the database had been backed up in a consistent state, a backup control file was used, and no further archive logs needed to be applied. 8. Re-create and size the Oracle temp files for the temporary tablespaces TEMP and TEMP1. The SQL commands used to do this are shown in Figure 29. Figure 29. Re-creating the Oracle temp files This completed the successful restore and recovery of the database and provided the first step in validating the integrity of the database. The total time taken for the restore and recovery operations was 27 hours, 8 minutes. 31

32 Verifying the integrity of the database Figure 30 shows a breakdown of the restored database objects and files and their mappings to the Oracle ASM disk groups. Figure 30. Mapping of Oracle database objects to Oracle ASM disk groups As part of the discovery process, EMC IT supplied the Cork team with Automatic Workload Repository (AWR) reports from the production database. A number of key SQL queries in the reports were identified to verify the integrity of the restored database. At the remote site, each of these queries was extracted from the AWR reports and run against the restored database. 32

33 Figure 31 demonstrates this process for one of the queries. The first item is an extract from the SQL ordered by Gets section of an AWR report. The second item is an extract from the AWR report for SQL ID: 0v7knzwvnc9u7. The third item shows the Oracle TKProf output of the trace file gathered for this SQL query when it was run against the restored database. SQL query ID SQL statement Figure 31. Testing the integrity of the restored database 33

34 Data Domain consolidation remote site Overview The Solutions Center in Cork uses a shared Data Domain DD880 for consolidated backup on many projects within the lab. To ensure consistency, and to maintain a copy of the EMC IT Production DB in its original state, Data Domain Replicator was used to copy the database image to the DD880 at the end of the project, as shown in Figure 32. The DD660 was then repurposed to another project. Cork Solutions Center EMC Data Domain DD660 DD Replicator EMC Data Domain DD880 Cork Solutions Center shared backup target Figure 32. Database consolidation to shared DD880 Data Domain Replicator software provides automated, fast, and reliable replication of data for disaster recovery (DR), remote office data protection, and multiple site tape consolidation. Replicator duplicates the compressed, deduplicated data over a network (including WANs), greatly reducing the demands on the network. Once replication has been configured between a source and a destination, any new data written to the source can be automatically replicated to the destination for the use case, a one-off copy was made to the DD880 but continued replication was not required. Replication process Setting up replication between the DD660 and DD880 involves simply creating a replication pair by specifying the replica source and destination. This can be done using the DD Enterprise Manager, as shown in Figure 33, or using the DD OS CLI. 34

35 Figure 33. Creating a replication pair The Replication view in DD Enterprise Manager provides configuration, status, performance, and topology information for the replication pair. Figure 34 shows this information for the use case replication process at completion. Figure 34. Data Domain Enterprise Manager Replication view 35

36 Figure 35 shows the timeline for the replication process, which took 23 hours to complete. Figure 35. Data Domain replication timeline 36

37 Conclusion Summary This white paper captures and describes the process used to transport a 12 TB Oracle data warehouse between two EMC data centers and to restore and recover the database at the target site. The data centers were located on different continents and over 4,000 kilometers apart. Data security was a critical requirement. If the transport mechanism was not secure, then it would not meet EMC corporate governance compliance and the move could not happen. In addition, as the process involved a real-world data warehouse, the solution had to be nondisruptive to EMC core production activities. The large size of the database and the long distance involved meant that: Transfer over a shared WAN link was neither practical nor efficient Creating a dedicated WAN link was neither practical nor economical In comparison, backing up the data to a Data Domain DD660 appliance, and transporting this by commercial carrier to the target site, proved to be a secure, practical, economical, and efficient solution. At the primary site, EMC TimeFinder enabled offloading of the backup process to a non-production server, and using an NFS mount allowed simple RMAN backup and restore on the DD660. Findings The key findings of this solution include: Nondisruptive backup EMC TimeFinder and seamless Data Domain integration with Oracle RMAN ensured zero impact to the production environment. Security and corporate governance compliance Data Domain data integrity, Data Domain Encryption, and Data Domain file system encryption locking secured the system, enabling it to be shipped by standard commercial transport. Simple restore and recovery Using an NFS mount allowed simple RMAN backup and restore from disk. Data Domain technology ensured balanced backup and restore speeds. The 12 TB data warehouse was available for use within 3 days of arrival in Cork. Economic transport Data Domain deduplication reduced the database storage and transport requirements from 12 TB to 3.3 TB a small form factor DD660 (2U, 60 kg) minimized shipping costs. 37

38 References White papers Product documentation Other documentation For additional information, see the white papers listed below. EMC IT's Migration to the Open, Expandable Oracle BI Grid Applied Technology EMC Backup and Recovery for Oracle Database 11g Data Warehouse A Detailed Review For additional information, see the product documents listed below. EMC DD OS 4.9 Administration Guide EMC Solutions Enabler Symmetrix CLI 7.2 Command Reference For additional information, see the documents listed below. Oracle Clusterware Administration and Deployment Guide 11g Release 1 (11.1) Oracle Real Application Clusters Administration and Deployment Guide 11g Release 1 (11.1) Oracle Clusterware Installation Guide 11g Release 1 (11.1) for Linux Oracle Database Installation Guide 11g Release 1 (11.1) for Linux Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for Linux and UNIX Oracle Database Backup and Recovery Reference 11g Release 1 (11.1) Oracle Database Backup and Recovery User's Guide 11g Release 1 (11.1) 38