Hitachi-Oracle s BCM Platform Solution Verification Report on Oracle Active Data Guard

Transcription

1 Hitachi-Oracle s BCM Platform Solution Verification Report on Oracle Active Data Guard Date: March 28 Version:

2 1. Introduction Oracle Database 11g Release 1 was released in October 27 as the latest major version of Oracle Database. In this version, Oracle Data Guard offers a number of new and innovative features to help ensure business continuity by protecting important corporate data, including a feature that initiates a failover to a remote standby system in the event the production system fails due to a disaster or emergency. Oracle Corporation Japan and Hitachi Ltd. performed verification tests of Oracle Data Guard at the Oracle GRID Center, building a large-scale transaction environment for a simulated production system combining Hitachi BladeSymphony high-reliability blade servers and Oracle Database 11g Release 1. This white paper introduces the BCM (Business Continuity Management) platform solution realized by combining Hitachi s hardware and Oracle Database 11g Release 1 and results of verification with respect to the effectiveness of features provided by Oracle Active Data Guard, a new option in the Oracle Database 11g Release

3 Acknowledgements Oracle Corporation Japan established a partnership with Hitachi Ltd. and other grid strategy partner companies in November 26, opening the Oracle GRID Center ( a facility that incorporates the most advanced technologies, with the goal of constructing next-generation business solutions capable of optimizing enterprise system infrastructures. Publication of this white paper was made possible by hardware and software provided to the Oracle GRID Center by Intel Corporation and Cisco Systems G.K., which support the purpose of the Oracle GRID Center, as well as support and aid provided by engineers from these companies. We wish to express our sincere gratitude to the companies and engineers for their support. *All rights reserved. Disclaimer This document is provided for informational purposes only. The contents hereof are subject to change without prior notice. Oracle Corporation Japan or Hitachi, Ltd does not warrant that this document is error-free, nor does it provide any other warranties or conditions, whether expressed or implied, including implied warranties and conditions of merchantability or fitness for a particular purpose. Oracle Corporation Japan and Hitachi Ltd. specifically disclaim any liability with respect to this document. No contractual obligations are formed by this document, either directly or indirectly. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without prior written permission from Oracle Corporation Japan and Hitachi Ltd. Trademarks BladeSymphony is a registered trademark of Hitachi Ltd. ORACLE is a registered trademark of Oracle Corporation. Intel and Xeon are trademarks of Intel Corporation in the United States and other countries. Red Hat is a trademark or a registered trademark of Red Hat Inc. in the United States and other countries. Linux is a registered trademark of Linus Torvalds. Cisco is a registered trademark of Cisco Systems, Inc. in the United States and other countries. Other names of companies and products used herein are trademarks or registered trademarks of their respective owners

4 2. Contents 1. Introduction Contents Criticality of Business Continuity Management (BCM) Oracle Data Guard Examples of BCM Platform Solutions Realized by Hitachi and Oracle Verifying Oracle Active Data Guard Purpose and specifics of verification tests Verification environment System configuration Hardware used Software used About workloads Verification Results Creating a standby database using RMAN network duplicate Effective use of standby site via Oracle Active Data Guard and reductions in system downtime based on effective use of standby site Measuring REDO apply performance for standby database Fast-Start Failover Failover under high-load transaction condition Summary

5 Figures Figure 4-1: Schematics of Oracle Data Guard operation... 7 Figure 4-2: Effective use of standby database via Real-time Query... 8 Figure 4-3: Effective use of standby database with Snapshot Standby... 8 Figure 4-4: Fast-Start Failover operation... 9 Figure 5-1: Online system maintenance based on Hitachi hardware and Oracle Data Guard... 1 Figure 5-2: Data protection with rapid application of server resources at reduced standby cost Figure 6-1: Configuration of the system used in verification tests Figure 7-1: Conventional standby database production method Figure 7-2: Creating a standby database using RMAN network duplicate Figure 7-3: Previous drawbacks Relationship between standby site use time and system downtimes... 2 Figure 7-4: Effective use of standby site via Oracle Active Data Guard Figure 7-5: Simulated business scenario used in verification tests Figure 7-6: Process of failover to physical standby database Figure 7-7: Low REDO apply performance Figure 7-8: Adequate REDO apply performance Figure 7-9: Fast-Start Failover operation Figure 7-1: Verifying failover under high-load transaction conditions Table Table 7-1: Apply performance comparison patterns Table 7-2: Verification configuration patterns Table 7-3: Verified failure patterns Table 7-4: Verified failure patterns and verification results... 3 Graphs Graph 6-1: CPU usage of primary database servers during load generation Graph 7-1: Comparison of standby data production times (via conventional method and using RMAN network duplicate) Graph 7-2: CPU usage and network transfer volume in creation of standby database via conventional method (top: primary database server, bottom: standby database server) Graph 7-3: CPU usage and network transfer volume in production of standby database using RMAN network duplicate (top: primary database server, bottom: standby database server) Graph 7-4: Business transaction throughput, CPU usage of primary database server, and network transfer volumes during creation of standby database using RMAN network duplicate19 Graph 7-5: Effective use of CPU resources of standby site with Oracle Active Data Guard Graph 7-6: Reductions in system downtime via Oracle Active Data Guard during use of physical standby site Graph 7-7: Comparison of volume of generated REDO against REDO apply performance Graph 7-8: Apply performance comparison Graph 7-9: Transactions during failure of all instances for the primary database and patterns in CPU usage for individual database servers

6 3. Criticality of Business Continuity Management (BCM) IT systems have grown increasingly important for corporations. Even in the event of an earthquake-induced site failure or system failure caused by hardware malfunction, corporations must continue to safeguard critical business data such as customer information and rapidly restore system functionality to ensure continuing services. In particular, corporations must meet the following requirements: Business continuity Interruptions or outages affecting important services pose serious threats to the entire business, in certain cases resulting not just in lost income, but serious damage to the confidence of customers and associated companies. Data protection Data remains a critical asset for any company. Corporate data for example, payroll or employee information, client records, valuable research results, financial records, or history information can require both significant sums and effort to reconstruct or regenerate once lost, if this is even possible, and in some cases such data loss may impair a company s capacity to continue operating. System flexibility to adapt to changes IT systems must ensure business continuity even in the event of unplanned system downtimes, including system failure. These systems must also minimize the duration of planned downtimes, including downtimes for software updates and hardware maintenance, to reduce any negative effects on business operations. Particularly in the case of open systems, the rapid pace of software development requires that procedures for updating software and applying software patches be kept as short as possible in order to keep systems up to date and maintain systems in a robust condition. With respect to hardware, rapid developments in multi-core CPU technology in recent years now makes it possible in certain cases to improve performance and reduce TCO simply by replacing existing equipment with the latest hardware. In general, agility and flexibility have become enterprise system requirements. Cost efficiency Effective use of standby sites Also important for ensuring high cost efficiency is effective use of the server resources at standby sites set aside for disasters and other emergency situations. Ensuring high cost efficiency leads to the acquisition of countermeasures against system failure. Low resource efficiency at established standby sites during ordinary operations, on the other hand, will generally make it more difficult to acquire adequate funding, etc. for systems. Combining Hitachi BladeSymphony or Hitachi Storage hardware with Oracle Real Application Clusters (Oracle RAC) and Oracle Data Guard makes it possible to deliver a solution that resolves such issues

7 4. Oracle Data Guard Oracle Data Guard creates a standby database as a copy of the production database (called the primary database) and provides features that perform a series of comprehensive services for that database, including maintenance, management, and monitoring. A standby database is created as a copy that maintains transactional consistency with the primary database. Following the creation of the standby database, REDO sent from the primary database are used to reflect changes made in the primary database. If the primary database becomes unavailable due to down, whether planned or unplanned, the standby database gains primary database status to minimize the downtime. The Oracle Data Guard is provided by Oracle Database Enterprise Edition. In normal operation In emergencies Primary database Primary database Standby database Standby database Copy Primary database connected during normal operation Connection switches to standby database in the event of failure. Figure 4-1: Schematics of Oracle Data Guard operation Standby databases generally come in one of two configurations. One, a physical standby database, is identical to the primary database at the physical block level. The other, a logical standby database, is identical to the primary database at the logical row data level. The version of Oracle Data Guard in Oracle Database 11g Release 1 features various enhancements. Introduced below are some of the new features examined in our verification testing. Oracle Active Data Guard In previous release versions, application of REDO had to be suspended when accessing data in a physical standby database. A Oracle Active Data Guard option with Oracle Database 11g Release 1 enables access to data in a physical standby database without suspending the application of REDO. This feature is called Real-time Query. This feature enhancement allows normal use of a physical standby database for reporting and other tasks

8 Normal operation Off -loading of reporting process and backup acquisition to standby database Patch process reporting Backup acquisition Oracle Data Guard Primary database Physical standby database Figure 4-2: Effective use of standby database via Real-time Query Oracle Active Data Guard features a high-speed incremental backup feature based on a change-tracking file when obtaining backups from a standby database, thereby offering both high availability and convenient data protection against failures in the event of planned downtimes or unplanned outages at the production site. Snapshot Standby The Snapshot Standby feature enables temporary use of a physical standby database as an easy-to-use read-write test database. Even while being used as a test database, the physical standby database can receive REDO from the primary database, allowing it to continue providing the data protection feature. A snapshot standby database is also easily returned to physical standby database status. Normal operation Open as a temporary readwrite test database Client for testing REDO transfers continue while database is open Oracle Data Guard Primary database Snapshot standby Figure 4-3: Effective use of standby database with Snapshot Standby - 8 -

9 Creating a standby database using RMAN network duplicate Previous release versions required the acquisition of a full backup of the primary database on local site, transfer of the backup to standby site and restoring of the backup to create a standby database. With Oracle Database 11g Release 1, the enhanced Recovery Manager (RMAN) network duplicate feature, used for database duplication, backups primary database while at the same time restoring over the network to the standby. Network duplicate saves time and storage Fast-Start Failover The Fast-Start Failover provides a feature that automatically detects failures in the primary database and initiates failover after failure detection. Detection of failure and initiation of failover are performed by the observer set up separately from the primary database and standby database. The observer is a component of Data Guard Broker. Fast-Start Failover enables automatic failover in the event of a primary database failure without administrator intervention. Monitoring Observer Monitoring Primary database REDO transfer Automatic failover Standby database Figure 4-4: Fast-Start Failover operation In previous release versions, Fast-Start Failover could be used only in Maximum Availability mode which required synchronous transfers of REDO. Oracle Database 11g Release 1 now supports Maximum Performance mode to allow asynchronous REDO transfer settings, allowing use in a wider range of operating environments. The new version also provides greater flexibility in determining whether or not to initiate a failover at the time of failure detection, thereby meeting various failover requirements

10 5. Examples of BCM Platform Solutions Realized by Hitachi and Oracle Described below are some examples of the BCM solution realized through the combination of Hitachi hardware and Oracle Database 11g Release 1. Online system maintenance Figure 5-1 shows an example of a Data Guard system configuration consisting of a production business environment and a test environment. The test environment is used for report tasks using Oracle Active Data Guard features or as a development environment using the Snapshot Standby feature. This sample configuration permits not only the application of patch sets to Oracle software and version updates, but also BladeSymphony server blade replacements and additions in combination with the Oracle Data Guard switchover feature, and seamless online disk addition to production environments via Hitachi Storage virtualization. The combination of Hitachi hardware and Oracle Database 11g Release 1 enables online maintenance of both software and hardware with minimal impact on production operations. (2) Replacement with new blade server Production environment Oracle rolling upgrades (1) Switchover to test environment Oracle Data Guard configuration Online blade server replacement Swi tchover of pr oduction environment to minimiz e impac t on business operations Online hard disk addition to storage pool No need to s et LVM, ASM, or other OS No need to reboot f or dis k recogniti on Test environment Figure 5-1: Online system maintenance based on Hitachi hardware and Oracle Data Guard Data protection at reduced standby costs and rapid addition of server resources Figure 5-2 shows an example of a configuration with minimum allocation of standby database server resources. It provides data protection using Oracle Data Guard while minimizing standby database costs. If the primary database fails due to a disaster or other reason, a failover to the standby database is initiated to enable continuing business operations. However, restoring the - 1 -

11 service levels of the primary database generally requires the allocation of additional resources to ensure the same level of processing capacity as the primary database a requirement that generally costs a great deal of time and money. But combining the provisioning features of BladeSymphony and Oracle Real Application Clusters can significantly reduce the cost of adding server resources while enabling immediate response. Primary database 4-node RAC Data Guar d configuration Standby database 1-node RAC Normal operations Maintaining data protection at low initial cost by allocating minimum server resources to the standby database Primary database failure due to disaster... Primary database failure Primary database 4-node RAC Data Guar d configuration Standby database 1-node RAC + 3 nodes Provisioning Additional server resources are required if the standby database is used to continue business operations. Combining BladeSymphony's and Oracle's provisioning functions enables significantly simplified additional tasks and immediate response. Figure 5-2: Data protection with rapid application of server resources at reduced standby cost

12 6. Verifying Oracle Active Data Guard 6-1 Purpose and specifics of verification tests We performed verification testing at the Oracle GRID Center with the following three main goals: Confirming the effectiveness of new Oracle Data Guard features We performed verification tests to confirm the effectiveness and usability of the new Oracle Data Guard features and to check for any important considerations when using the features. In the verification testing, we focused mainly on the following features: Creating a standby database using RMAN network duplicate Benefits of creating a standby using RMAN network duplicate feature Oracle Active Data Guard Benefits of effectively using the standby database with Real-time Query feature of Oracle Active Data Guard and reductions in system downtimes based on effective use of the standby database Snapshot Standby Fast-start Failover Performance and failover under large-scale high-volume transaction We performed the verification tests to check for fast, effective failover to the standby database in the event of a failure while the primary database was under heavy loads and with the CPU and network resources at maximum capacity. Another goal was to identify any potential issues associated with use in large-scale, high-volume transaction environments. These represent critical performance aspects, since the primary purpose of introducing Oracle Data Guard is to achieve switchover to the standby site in the event of a primary site failure. Establishing best practices We performed verification testing to establish procedures for creating a standby database and managing an Oracle Data Guard environment. * For a list of the procedures that proved effective in our verification tests, please refer to the separate document titled Oracle Database 11g Release 1 Physical Standby Setting Guide(Japanese only)

13 6-2 Verification environment System configuration Figure 6-1 shows the configuration of the system used in our verification tests. The same public network was used to connect client machines to the database server and to transmit REDO from the primary site to the standby site. The network bandwidth was 1 Gbps. Client machines Cisco Catalyst 654 Cisco Catalyst 375 Database server: Hitachi BladeSymphony BS32 Primary site: 2-node RAC Standby site: 2-node RAC Storage: Hitachi Adaptable Modular Storage Primary site Standby site Figure 6-1: Configuration of the system used in verification tests Hardware used Database server Model CPU Memory Hitachi BladeSymphony BS32 4 blades Dual-Core Intel Xeon processor 3 GHz 2 sockets/blade 8 GB Client machine Model CPU Memory Intel White Box, 4 units Quad-Core Intel Xeon processor 2.66 GHz 1 socket/server 4 GB Storage Model Hard disk RAID group configuration Hitachi Adaptable Modular Storage (AMS) 144 GB 28 HDD (+ 2 HDD as spare) 2D+1P 8 (for Oracle database)

14 6-2-3 Software used Database server OS Red Hat Enterprise Linux 4.5 Oracle Oracle Database 11g Release 1 ( ) Enterprise Edition Oracle Real Application Clusters Oracle Active Data Guard Oracle Partitioning Client machine OS Red Hat Enterprise Linux 4 Update 3 Oracle Oracle Client 1g Release 2 (1.2) About workloads In our verification tests, we used an online transaction processing system (OLTP) for a simulated online Web shopping site as a workload model. SQL statements generated by JPetStore were provided as a sample application for Spring Framework ( an open-source J2EE framework, were multi-executed by a custom application. The process flow is described below. (1) User sign-on A user ID was randomly selected and a search performed for user information. select from account, profile, signon where account.userid=? and signon.password =? and ; (2) Product search A keyword for product search was randomly generated and a search performed for the product. Adjustments were made so that the search results totaled approximately 1 on average. select from category where catid =?; select from product where(lower(name)like?); (3) Product selection One item was selected from the search results (hits). select from item, product where i.itemid =? and (4) Stock quantity check The quantity of the selected item in stock was checked. select from inventory where itemid =? (5) Order placement Order data for the specified product was issued. insert into orders ; insert into orderstatus ; insert into lineitem ; The quantity of ordered products was subtracted from the inventory quantity in the stock management list. Update inventory set qty=qty-1 where itemid =?; (6) Order finalization commit

15 The above-mentioned processes were multi-executed by client machines. As shown in Graph 6-1, the workload generated a heavy load on the primary database server. CPU usage of primary database server 1 user system iowait CPU usage of primary database server 2 user system iowait 1 1 CPU usage (%) CPU usage (%) Time (sec) Time (sec) Graph 6-1: CPU usage of primary database servers during load generation 7. Verification Results 7-1 Creating a standby database using RMAN network duplicate Creating a standby database requires the copying of database files from the primary database to the standby site. With versions up to Oracle Database 1g, this was generally achieved by obtaining a backup of the primary database and transferring backup files to the standby site via network using ftp or scp, or by copying the backup file to a tape and sending the tape to the standby site. Oracle Database 11g Release 1 enhances the RMAN duplicate command to allow copying of database files from the primary database currently online directly to the standby site. This eliminates the need to obtain a backup at the primary site and to produce a duplicate from the backup at the standby site. It also eliminates the need to arrange a disk space to store the backup file at both the primary and standby sites. Comparison and verification of standby database creation by the conventional method and using RMAN network duplicate We created standby databases by the conventional method and from the active database, measuring the time required to create a standby database and CPU usage during that process. We then compared and examined the results. The total size of the primary database used in this verification test was approximately 17 GB. Conventional method (Figure 7-1) (1) A backup file was created by online backup using RMAN (2) The backup file was sent from the primary site to the standby site across a network using scp. (3) The database was restored from the backup file by RMAN

16 Creating a standby database from an active database (Figure 7-2) (1) The online primary database file was copied directly to the standby database. Conventional standby database construction method Primary database Standby database (1) Creating a backup file (online backup by RMAN) Backup file Backup file (3) Database restored by backup file using RMAN. (2) Transfer of backup file by scp Primary site Standby site Figure 7-1: Conventional standby database production method Creating a standby database using RMAN network duplicate Primary database Standby database (1) Directly copying an online database file Primary site Standby site Figure 7-2: Creating a standby database using RMAN network duplicate Graph 7-1 compares the time required to create a standby database by the conventional method and directly from the active database. Creating a standby database from the active database does not require the creation of a backup at the primary site and the restoration of the database at the standby site, enabling creation of the standby database in about 1/3 the time required by the conventional method

17 Conventional method Creating a standby database using RMAN networ k duplic ate Time (sec) Graph 7-1: Comparison of standby data production times (via conventional method and using RMAN network duplicate) Graph 7-2 shows the CPU usage of the primary database server and standby database server and network transfer volumes during the creation of the standby database by the conventional method. Approximately 3% of the CPU resources were used to create a backup file at the primary site and to restore the database at the standby site. Graph 7-3 shows the CPU usage of the primary database server and standby database server and network transfer volumes during the creation of the standby database from the active database. Compared to the conventional method, creating a standby database from the active database kept CPU usage at low levels and achieved efficient network transfer/copying of online data files. And network transfer volumes per unit time are high, resulting in higher speeds than copying by scp. CPU usage (%) CPU usage of primary databas e s erver user system iowait Online backup by RMAN Backup file transfer by scp Time (sec) Network transfer volume (Kbytes/s) Networ k transfer vol ume of pri mary databas e s erver Receiving volume (kb/s) Backup file transfer by scp Transmitting volume (kb/s) Time (sec) CPU usage of standby database server user system iowait Networ k transfer vol ume for s tandby databas e s erver Receiving volume (kb/s) Transmitting volume (kb/s) CPU usage (%) Database restoration by RMAN Time (sec) Network transfer volume (Kbytes/s) Backup file reception by scp Time (sec) Graph 7-2: CPU usage and network transfer volume in creation of standby database via conventional method (top: primary database server, bottom: standby database server)

18 CPU usage of primary databas e s erver user system iowait Net wor k transfer vol ume of pri mary databas e s erver Receiving volume (kb/s) rxkb /s Transmitting txkb /s volume (kb/s) CPU usage (%) Direct copying of online database file Time (sec) Network transfer volume (Kbytes/s) Time (sec) CPU usage of sec ondar y database server Net wor k transfer vol ume of sec ondar y dat abase server user system iowait Receiving volume (kb/s) rxkb /s Transmitting txkb /s volume (kb/s) CPU usage (%) Time (sec) Network transfer volume (Kbytes/s) Time (sec) Graph 7-3: CPU usage and network transfer volume in production of standby database using RMAN network duplicate (top: primary database server, bottom: standby database server) Effect on business transactions during creation of standby database using RMAN network duplicate To examine the effects on business transactions of creating a standby database while transactions are being processed, we created a standby database from the active database while generating a business transaction load on the primary database. Graph 7-4 shows results for measurements of business transaction throughput, CPU usage of the primary database server, and network transfer volumes. In this case, contention between business transaction processing and database file transfer processing reduced business transaction throughput by approximately 2%. Transfer volumes of nearly 8 MB/s were recorded during the transfer of the database file. Since business transactions under ordinary operating conditions utilized approximately 2 MB/s, database file volumes transferred to the standby site are estimated to be about 6 MB/s. Since transfer volumes would be lower than under conditions with no load, it took longer to create a standby database in this test case. The effect on the business transaction performance is expected to vary depending on the process characteristics of the transaction being processed. In actual use, we recommend that users consider creating a standby database in a time with low business loads to minimize effects on business operations, as well as configuring a separate network to transfer REDO. In high latency network environment like WAN, throughput of network duplicate might be improved by tuning network I/O buffer size. Please refer to '14.2 Configuring I/O buffer space' of Net Services Administrator s Guide 11g Release 1(11.1)

19 Transaction throughput Transaction throughput Time (sec) Standby databas e production in process Effect of creation of standby dat abase using RMAN net wor k duplicat e on tr ans action t hroughput was about 2% i n our verific ation tests. Total transf er vol ume was about 8 MB/s. By subtrac ting about 2 MB/s us ed by busi ness trans actions fr om t his figure, t he dat abase file trans fer vol ume is estimated t o be about 6 MB/s. CPU usage (%) CPU usage of primary database server user system iowait Time (sec) Network transfer volume (Kbytes/s) Network transfer volume of primary database server Receiving volume rxkb (kb/s) Transmitting txkb volume /s (kb/s) Time (sec) Graph 7-4: Business transaction throughput, CPU usage of primary database server, and network transfer volumes during creation of standby database using RMAN network duplicate 7-2 Effective use of standby site via Oracle Active Data Guard and reductions in system downtime based on effective use of standby site Oracle Data Guard versions up to Oracle Database 1g had the following issue related to effective use of the standby site. Application of REDO had to be stopped when the standby site is used on a read-only basis by physical standby features. A periodic data synchronizing process was required to reduce downtimes caused by primary site failure. This meant the standby site had to be set to the managed recovery mode at regular intervals, making operations more complicated. Logical Standby are accessible during application of REDO, but there are limitations relate to the data type and other factors. These restrictions meant using the standby site previously required complex procedures. Longer standby site use times meant longer times required to recovery the database in case of failure, impairing availability (Figure 7-3)