Oracle/HP Best Practices Guide for HP IO Accelerators

Transcription

1 Oracle/HP Best Practices Guide for HP IO Accelerators Part Number December 2010 (First Edition)

2 Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR and , Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor s standard commercial license. Microsoft, Windows, and Windows Server are U.S. registered trademarks of Microsoft Corporation. Oracle is a registered trademark of Oracle and/or its affiliates. Intended audience This document is for the person who installs, administers, and troubleshoots servers and storage systems. HP assumes you are qualified in the servicing of computer equipment and trained in recognizing hazards in products with hazardous energy levels.

3 Contents Overview... 5 Introduction... 5 Single-instance and RAC overview... 5 Using IO Accelerators with Oracle... 5 Setting up filesystems... 6 Using ASM... 6 Redundancy architectures... 7 Redundancy best practices... 7 Filesystem hosted data files... 7 ASM Mirroring... 7 External redundancy (striping/raid 0)... 7 Normal redundancy (mirroring/raid 1)... 7 High redundancy (mirroring/raid 10+1)... 8 ASM mirroring with a read-preferred device... 9 Using data guard... 9 Data Guard replication types Data Guard recommended reading Data Guard recommended tunings Generic performance tuning Write performance and steady state Separating data types Discovering your application I/O profile and selecting an architecture Using Oracle tools Oracle Enterprise Manager Single-instance performance architectures Hosting all data areas on the IO Accelerator Hosting temp space on the IO Accelerator Example A: Migrating temp tablespace to an IO Accelerator on a filesystem Example B: Migrating temp tablespace to an ASM disk group backed by IO Accelerators Hosting redo logs on the IO Accelerator Example C: Migrating redo logs to an IO Accelerator on a local filesystem Example D: Migrating redo logs to an IO Accelerator configured with ASM Using the IO Accelerator with flash cache Prerequisites Assessing flash cache performance improvements Setting up flash cache Enabling flash cache Sizing the flash cache Tuning memory for the flash cache Using flash cache Example: Flash cache use Oracle RAC performance architectures SRP and iser targets Contents 3

4 Acronyms and abbreviations Contents 4

5 Overview Introduction This document explains best practices for integrating HP IO Accelerator technology in Oracle environments. The three main sections are: Redundancy architectures (how to keep your data safe and available) Performance tuning for the demands of your Oracle database Performance architectures (how to use the IO Accelerator to speed up Oracle to meet your reliability, performance, and consolidation needs) Significant performance improvements can be attained by using IO Accelerators, instead of conventional storage, even if no major I/O bottlenecks occur in the existing storage. These improvements in throughput are typically the result of the IO Accelerator low latency. Many Oracle databases target 5-15 milliseconds as an acceptable I/O latency time, but the IO Accelerator can deliver sub 1millisecond access times, even for multi-i/o transaction groups. The redundancy section offers many options for providing data protection. Because every deployment needs a different level of data reliability guarantees, reading through the options in this section is highly recommended. The performance characterization section helps you identify which areas in your Oracle database are currently bottlenecks, and it makes recommendations on how to use IO Accelerators to alleviate those bottlenecks. The performance architectures section contains the details of various IO Accelerator deployment options that can be used to alleviate the bottlenecks found through performance characterization. It also explains how to migrate data off existing storage and on to IO Accelerator-backed storage. Single-instance and RAC overview Single-instance Oracle runs on a single server is the most common type of Oracle deployment. For information on using IO Accelerators to enhance system performance on single-instance Oracle systems, see "Single-instance performance architectures (on page 16)." Oracle RAC enables a cluster of servers to operate on the same database. This provides a highly faulttolerant environment, where the loss of any individual server, or even multiple servers, does not impact the availability of the database to the application. For more information on Oracle RAC, see the Oracle website ( ). Using IO Accelerators with Oracle Several options are available for using IO Accelerators to host Oracle storage needs. The IO Accelerators can be formatted with a standard filesystem, added to ASM volumes, or set as flash cache targets. Overview 5

6 For more information on using flash cache with Oracle databases, see "Using the IO Accelerator with flash cache (on page 22)." Setting up filesystems Oracle can host its data files on any standard filesystem built on IO Accelerators. Typical configurations include aggregating drives using RAID or LVM for performance or redundancy. See the IO Accelerator User Guide for your operating system for additional recommendations for filesystems, RAID setup, and filesystem tuning. Using ASM The IO Accelerator behaves just like any block device and can be managed with ASM in the same manner as any other direct-attach storage. Benefits of ASM include enhanced manageability and the leveraging of data replication features. At this time, a known issue exists with configuring ASM with IO Accelerators. For details, see the Oracle ASM Support page ( For more information on using flash cache with Oracle databases, see the examples in the Hosting Logs on the IO Accelerator section later in this document. Overview 6

7 Redundancy architectures Redundancy best practices When a failover system is not available to provide failure protection, all storage devices which are not highly-available, including IO Accelerators, must be mirrored to provide redundancy. Local redundancy is not as critical for designs that include both off-server replication and a hot standby failover server ready to take over. However, local redundancy might still be used in some environments. Filesystem hosted data files If you use a filesystem to host the database data files, and no off-server replication is being used, HP recommends that RAID 1 or RAID 10 be used over the top of the IO Accelerator. For instructions, best practices, and tuning recommendations for setting up a RAID device and filesystem across multiple IO Accelerators, see the HP IO Accelerator User Guide for your operating system. ASM Mirroring The following sections discuss various mirroring configurations. External redundancy (striping/raid 0) In a RAID 0 configuration, ASM does not provide mirroring redundancy. It relies on the storage system to provide any needed data protection. ASM can aggregate multiple IO Accelerator block devices into a single disk group. All disks must be located to successfully mount the disk group. The following command creates a disk group with four 320GB IO Accelerators, fioa through fiod, resulting in a 1280GB disk group: ASM SQL > CREATE DISKGROUP DATADG EXTERNAL REDUNDANCY DISK '/dev/fio[a-d]'; The following command creates a second disk group of four 320GB drives, fioe through fioh, each down-formatted to 200GB for a total disk group size of 800GB. ASM SQL > CREATE DISKGROUP LOGDG EXTERNAL REDUNDANCY DISK '/dev/fio[e-h]'; Normal redundancy (mirroring/raid 1) ASM provides two-way mirroring. By default, all files are mirrored so that every data extent has two copies. A loss of one ASM disk in any failure group participating in the mirror is tolerated. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all the devices. This example demonstrates the creation of two ASM disk groups by mirroring the following device pairs: /dev/fioa to /dev/fioc Redundancy architectures 7

8 /dev/fiob to /dev/fiod /dev/fioe to /dev/fiog /dev/fiof to /dev/fioh The following command mirrors four devices (fioa through fiod) to create the +DATADG disk group. ASM SQL > CREATE DISKGROUP DATADG NORMAL REDUNDANCY FAILGROUP failure_group_1 DISK '/dev/fioa' NAME diska1, '/dev/fiob' NAME diska2, FAILGROUP failure_group_2 DISK '/dev/fioc' NAME diskb1, '/dev/fiod' NAME diskb2; The following command mirrors across four devices (/dev/fioa[e,f,g, h]) to create the LOG-DG disk group. ASM SQL > CREATE DISKGROUP LOGDG NORMAL REDUNDANCY FAILGROUP failure_group_1 DISK '/dev/fioe' NAME diska1, '/dev/fiof' NAME diska2, FAILGROUP failure_group_2 DISK '/dev/fiog' NAME diskb1, '/dev/fioh' NAME diskb2; High redundancy (mirroring/raid 10+1) ASM can also mirror data to three failure groups. This provides an additional level of redundancy above the normal setting, allowing failure of up to two disks in different failure groups to be handled gracefully. The effective disk space in a high-redundancy disk group is one-third the sum of the disk space in all the devices. The following command mirrors devices /dev/fio[a,c,e] and then mirrors devices /dev/fio[b,d,f] and stripes across the resulting two mirrored volumes, creating the DATADG disk group. ASM SQL > CREATE DISKGROUP DATADG HIGH REDUNDANCY FAILGROUP failure_group_1 DISK '/dev/fioa' NAME diska1, '/dev/fiob' NAME diska2, FAILGROUP failure_group_2 DISK '/dev/fioc' NAME diskb1, '/dev/fiod' NAME diskb2, FAILGROUP failure_group_3 DISK '/dev/fioe' NAME diskc1, '/dev/fiof' NAME diskc2; Redundancy architectures 8

9 This command mirrors across three devices (/dev/fioa[g,h,i]) to create the LOGDG disk group. ASM SQL > CREATE DISKGROUP LOGDG HIGH REDUNDANCY FAILGROUP failure_group_1 DISK '/dev/fiog' NAME diska1, FAILGROUP failure_group_2 DISK '/dev/fioh' NAME diskb1, FAILGROUP failure_group_2 DISK '/dev/fioi' NAME diskc1, ASM mirroring with a read-preferred device ASM enables you to select a device in a mirrored disk group that handles all the reads and receives all the writes. This enables the other devices in the group to receive only the writes. Using this solution, the IO Accelerator can be paired in a mirror with traditional storage and maintain its significant performance advantage. However, log commit/sync write latency is still as slow as the slowest device in the pair. To configure this feature, set the ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter in the ASM instance to specify a list of failure group names as preferred read disks. Set the parameter where diskgroup is the name of the disk group and failuregroup is the name of the failure group, separating these variables with a period. ASM_PREFERRED_READ_FAILURE_GROUPS = diskgroup.failuregroup,... For example, to enable read preferred mode on DATADG.failure_group_1 use the following command: ASM_PREFERRED_READ_FAILURE_GROUPS=DATADG.failure_group_1 The DATADG disk group could then be configured to include all IO Accelerators in Failure_group_1 and all storage devices that are not IO Accelerators in Failure_group_2 as follows: ASM SQL > CREATE DISKGROUP DATADG NORMAL REDUNDANCY FAILGROUP failure_group_1 DISK '/dev/fioa' NAME diska1, '/dev/fiob' NAME diska2, FAILGROUP failure_group_2 DISK '/dev/sdd' NAME diskb1, '/dev/sdc' NAME diskb2, In this example, it is assumed that each /dev/sd* device is as large as each /dev/fio* device. For more information on ASM, see the Oracle website ( ). Using data guard Oracle Data Guard provides the management, monitoring, and automation software to create and maintain one or more standby databases. These features help to protect Oracle data from failures, disasters, human error, and data corruption. Redundancy architectures 9

10 Administrators can use either manual or automatic failover to a Data Guard standby database to maintain high availability for mission-critical applications. Oracle Data Guard provides a primary/secondary solution (only one host can read and update the database at a time) rather than the primary/primary solution provided by RAC (each server can read and update simultaneously). Depending on the Oracle version and the type of standby database selected, the secondary host can be used for read-only access to the database while still functioning in a standby database role. When configuring replication with Data Guard, make sure you select the proper Protection Mode for your needs. The Protection Mode setting controls how Data Guard and Oracle treat the reporting of committed transactions to the client, based on the client's state on the secondary device. Options include both sync and async, as well as other nuances. With the sync option, transactions must be committed on both servers before processing can continue. With the async option, transactions must be committed only on the primary server to enable processing to continue. They are sent to the secondary server a short time later. The async option can allow for a small amount of data loss on failure of the primary server. For additional details on Protection Mode settings, see the Oracle website ( Data Guard replication types The log_archive_dest_n parameter in a Data Guard configuration supports a number of attributes including SYNC, ASYNC, ARCH and LGWR. NOTE: The ARCH and LGWR attributes have been depreciated in 11g Release 2. Specify SYNC or ASYNC. The ASYNC attribute is the default if neither attribute is specified. SYNCH and ASYNC These attributes specify whether the synchronous (SYNC) or asynchronous (ASYNC) redo transport mode is to be used. The LOG_ARCHIVE_DEST_11 through LOG_ARCHIVE_DEST_31 parameters do not support the SYNC attribute. The redo log data generated by a transaction that is configured with the SYNC attribute must have been received by every enabled destination before that transaction can commit. The redo data generated by a transaction that is configured with the ASYNC attribute need not have been received at a destination before that transaction can commit. This is the default behavior if neither SYNC nor ASYNC is specified. ARCH and LGWR These attributes specify whether redo transport services should use the archiving process (ARCH) or the log writer process (LGWR) to collect transaction redo data and transmit it to standby destinations. These attributes are optional. If neither the ARCH nor LGWR attribute is specified, the default is ARCH. Redo transport services use ARCn processes when the ARCH attribute is specified, and the log writer process when the LGWR attribute is specified. By default, archiving is performed by ARCn processes. You must explicitly specify the LGWR attribute for redo transport services to use the LGWR process. Although you cannot specify both LGWR and ARCn processes for the same destination, you can choose to use the log writer process for some destinations, while using the archiver process to transmit redo log data for other destinations. Redundancy architectures 10

11 If you change the current archival process of a destination (for example, from the ARCn process to the LGWR process), archival processing does not change until the next log switch occurs. Data Guard recommended reading For more information on Oracle 11g Data Guard Concepts and Administration, see the following references: Additional information on Data Guard ( Data Guard Overview ( html) Creating a Physical Standby Database ( Creating a Logical Standby Database ( ) Data Guard recommended tunings Initialization Parameters ( LOG_ARCHIVE_DEST_n Parameter Attributes ( Redundancy architectures 11

12 Generic performance tuning Write performance and steady state Solid state storage solutions often have poor right performance. This condition is caused by the characteristics of the underlying storage media, NAND flash. As more data is written to a solid state device, you must defragment the data on the device to continue operating. This process is called grooming. When the groomer starts, the IO Accelerator (like other solid state storage) experiences a reduction in sustained write performance and an increase in latency. This behavior is known, and the industry solution is trim (also known as discard). Trim enables a block device consumer (typically a filesystem) to notify the underlying device (such as an IO Accelerator) that some portion of the data is no longer needed. The device can use this information to improve grooming efficiency, thus improving the performance of the device while in steady state. But trim is a new feature and has not yet been deployed on many enterprise operating systems. Applications such as Oracle that allocate large files are not able to take advantage of trim, even when deployed on an operating system that supports it. Because of tight integration with the system, the IO Accelerator also provides a capability called underformatting. This feature is not available in solid state devices built on traditional storage protocols. By under-formatting an IO Accelerator, you can trade the presented capacity (for example, take a 160GB drive and present it as a 140GB drive) to achieve higher steady state write performance. For high-write workloads, this option provides a combination of usable capacity and write performance, rather than purchasing unnecessary capacity to achieve needed write performance. Because of the write-intensive nature of databases, even in read-heavy environments, HP recommends under-formatting to 70-80% of standard capacity as a good starting point for most Oracle deployments. Under-formatting can be done through the HP IO Accelerator Management Tool or through the commandline utility fio-format. For more details on formatting, see the User Guides for your operating system. Separating data types When hosting all or a combination of temporary, primary data area, and redo logs on IO Accelerators, you can achieve additional performance by isolating those areas to individual drives. This benefit can be accomplished by identifying the needed capacities and appropriate level of under-formatting for each storage area. For example: 1. Under-format the IO Accelerators to the appropriate level. 2. Create unique ASM disk groups/failure groups from the target IO Accelerators for each of the storage areas. You can take a similar approach if you are are using MD/LVM for device aggregation along with filesystems for hosting the data. Then the appropriate level of under-formatting must be applied to each target area, based on the workload. Generic performance tuning 12

13 Discovering your application I/O profile and selecting an architecture Using Oracle tools Understanding how your database uses its storage resources is the first step in selecting an appropriate architecture and applying any needed performance tunings. You can identify where the I/O bottlenecks are in your system. Then you can choose from a list of recommended architectures to help eliminate the bottlenecks. Oracle Enterprise Manager Determining what causes an I/O bottleneck in Oracle might be challenging. The Oracle tools you can use to accomplish this analysis are EM and the AWR and ADDM reports. With EM, you can easily detect I/O issues by looking for sessions or SQL statements that have high I/O waits. You can use EM to drill down into these sessions to look at the underlying SQL execution plans, or you can invoke an AWR report from the EM interface. Running AWR from EM provides further opportunity for drill-down analysis. You can also create the AWR and ADDM reports from the SQL command line, as described in the following sections. Running an AWR report You can generate an AWR for a database from EM or from the SQL command line, by using the awrrpt.sql SQL script. Beginning with Oracle 11g R2, you can use the awrgrpt.sql SQL script to create an AWR report that includes all available database instances in an Oracle RAC environment. Optionally, you can use the awrgrpti.sql SQL script to create the AWR report for a single instance or a combination of instances of the RAC database. These scripts can be found in the $ORACLE_HOME/rdbms/admin directory. The AWR report script generates an HTML or text report that displays statistics for a range of snapshot IDs. NOTE: When creating the AWR report, HP recommends selecting the HTML option. To generate an AWR report: 1. At the SQL prompt, enter the following 2. Specify whether you want an HTML or a text report. For example, enter the value for report_type as HTML. 3. Specify the number of days for which you want to list snapshot IDs. 4. Enter 2 as the value for num_days. A list of existing snapshots for the specified time range appears. Discovering your application I/O profile and selecting an architecture 13

14 5. Specify a beginning and ending snapshot ID for the workload repository report. For example, enter 150 for begin_snap and 160 for end_snap. 6. Enter a report name, or accept the default report name. For example, enter awrrpt_1_150_160 (the default) for report_name. If a text report had been specified, the awrrpt_1_150_160.txt report name is generated. Simple analysis of an AWR report Even if I/O bottlenecks are not found on your system, you can still improve performance by improving I/O time. The AWR report is divided into two sections. Section one consists of the following items: A description of the Oracle database configuration The range of snapshots being reported A summary of the major statistics for the reporting period: Load Profile, Instance Efficiency Percentages, Shared Pool Statistics, Host CPU, Instance CPU, and Memory Statistics Section two consists of the Main Report, which includes: Wait Event Statistics SQL Statistics, Instance Activity Statistics IO Stats Buffer Pool Statistics Advisory Statistics Wait Statistics Undo Statistics Latch Statistics Segment Statistics Dictionary Cache Statistics Library Cache Statistics Memory Statistics Streams Statistics Resource Limit Statistics Shared Server Statistics Initialization Parameters For example, to determine if db file sequential read is a top wait event, navigate to the Foreground Wait Events subsection of the Wait Events Statistics section. The corresponding SQL code responsible for the wait can be identified in the SQL Ordered by User I/O Wait Time subsection of the SQL Statistics section. Discovering your application I/O profile and selecting an architecture 14

15 When the top waits events in the AWR report are db file sequential read or db file scattered read, this is an indication of I/O waits. What is also important with these wait events is how long they are taking, on average. A sequential I/O (db file sequential read) takes 1-5 milliseconds. A random read (db file scattered read) takes up to milliseconds. However, even the typical access times can be greatly improved by using IO Accelerators, which also greatly reduces latency if db file scattered read conditions are encountered. Another factor with these wait events is the total read time. When analyzing the AWR report, you might observe some queries that perform millions of reads. For example, if a query is doing 10,000,000 sequential reads at 2 milliseconds each, this equates to 10,000 seconds or 2.7 hours of reading time. By using flash cache or hosting database data on IO Accelerators to reduce read time to less to 1 millisecond, you can save database access time. Even if an I/O bottleneck does not occur, you can improve performance by improving I/O time. For more information on using IO Accelerators and flash cache with Oracle databases, see "Singleinstance performance architectures (on page 16)." Running an ADDM report To analyze the database file sequential read wait event, an ADDM report can be generated. An ADDM analysis can be performed on a pair of AWR snapshots and a set of instances from the same database. The pair of AWR snapshots defines the time period for analysis, and the set of instances define the target for analysis. ADDM analysis is performed top-down, first identifying symptoms, and then refining them to reach the root causes of performance problems. The goal of the analysis is to reduce a single throughput metric called database time. Database time is the cumulative time spent by the database in processing user requests. It includes wait time and CPU time of all non-idle user sessions. By reducing database time, the database is able to support more user requests using the same resources, which increases throughput. The problems reported by ADDM are sorted by the amount of database time they are responsible for. System areas that are not responsible for a significant portion of database time are reported as non-problem areas. ADDM considers the following types of problems: CPU bottlenecks Undersized memory structures I/O capacity issues High-load SQL statements High-load PL/SQL execution and compilation RAC-specific issues Database configuration issues Concurrency issues Hot objects and top SQL for various problem areas In addition to problem diagnostics, ADDM recommends possible solutions. ADDM analysis results are represented as a set of findings. The scripts to generate this report are found in the $ORACLE_HOME/rdbms/admin directory and are named addmrpt.sql and addmrpti.sql. AWR and ADDM are part of the Automatic Database Diagnostic Monitoring features that were introduced in Oracle Database 10g. Discovering your application I/O profile and selecting an architecture 15

16 Single-instance performance architectures Hosting all data areas on the IO Accelerator This configuration is likely to be the highest performing of all proposed solutions. It is also likely to be the simplest to deploy, depending on your data protection needs. To create this configuration: 1. Place a filesystem on the IO Accelerator, or add the IO Accelerator devices to ASM in a configuration that includes the appropriate level of protection for your data. For example, you could use RAID1 or RAID10 or ASM mirroring of the IO Accelerators, as well as off-server replication such as a Data Guard periodic backup. 2. Use the IO Accelerator-hosted ASM groups or filesystems like any other block device. For more information on setting up this architecture, see "Generic Performance Tuning (on page 12)." Hosting temp space on the IO Accelerator This solution moves all temp space I/O off to local IO Accelerators while keeping the database and redo log file areas on traditional storage devices. Performance benefits, such as reduced query time, are seen most strongly in reporting or query-heavy environments dealing with large datasets. These types of datasets include: High-volume OLTP applications Query-intensive data warehouses Demanding internet applications This architecture might also benefit systems that are currently pushing their legacy storage near the limit. The benefit comes by offloading all temp space I/O, which frees up additional headroom on the legacy storage targets. Example A: Migrating temp tablespace to an IO Accelerator on a filesystem For this example: All of the current database datafiles reside in the directory /u01/oradata/testdb Device /dev/md0 represents a RAID 0 configuration of the /dev/fioa and /dev/fiob IO Accelerator devices /dev/md0 is configured to mount on /u02/oradata Final Filesystem Configuration To achieve the final filesystem configuration: 1. Change the ownership and permissions for the /u02/oradata directory to the following: chown -R oracle.oinstall /u02/oradata Single-instance performance architectures 16

17 chmod 775 /u02/oradata 2. Log on as the oracle user: su - oracle 3. Create the subdirectory for the TESTDB database under /u02/oradata: mkdir -p /u02/oradata/testdb The IO Accelerator is configured as an EXT3 filesystem and is ready for use by the Oracle TESTDB database. 4. As the oracle user, move the TEMP tablespace to the IO Accelerator. Migrating the TEMP Tablespace 1. Log on to sqlplus as sysdba: export ORACLE_SID=TESTDB sqlplus / as sysdba The TEMP tablespace cannot be moved during the mount stage by using the ALTER DATABASE RENAME FILE command. A workaround to this issue is to create a new temp tablespace. 2. Create a new temporary tablespace named TEMP2: CREATE TEMPORARY TABLESPACE TEMP2 TEMPFILE '/u01/app/oradata/temp201.dbf' SIZE 10240M AUTOEXTEND ON EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M; 3. Assign the default TEMP tablespace for the database to the TEMP2 tablespace: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp2; 4. Drop the old TEMP tablespace: DROP TABLESPACE temp INCLUDING CONTENTS AND DATAFILES; 5. Recreate the TEMP tablespace on the IO Accelerator filesystem: CREATE TEMPORARY TABLESPACE TEMP TEMPFILE '/u02/app/oradata/temp01.dbf' SIZE 10240M AUTOEXTEND ON EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M; 6. Assign the default TEMP tablespace for the database back to the TEMP tablespace: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp; 7. Drop the TEMP2 tablespace: DROP TABLESPACE temp2 INCLUDING CONTENTS AND DATAFILES; The TEMP tablespace now resides on the IO Accelerator. Example B: Migrating temp tablespace to an ASM disk group backed by IO Accelerators For this example: All of the current database data files reside in the +DATADG ASM disk group. The DB_CREATE_FILE_DEST parameter for the TESTDB database is +DATADG. An appropriately configured ASM disk group named +TEMPDG has been created and includes only the target IO Accelerators (likely using normal redundancy or some form of replication). Single-instance performance architectures 17

18 The TEMP tablespace cannot be moved during the mount stage by using the ALTER DATABASE RENAME FILE command. A workaround to this issue is to create a new temp tablespace. 1. Create a new temporary tablespace called TEMP2: CREATE TEMPORARY TABLESPACE TEMP2 SIZE 10240M AUTOEXTEND ON; 2. Assign the default TEMP tablespace for the database to the TEMP2 tablespace: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp2; 3. Drop the old TEMP tablespace: DROP TABLESPACE temp INCLUDING CONTENTS AND DATAFILES; 4. Recreate the TEMP tablespace in the IO Accelerator diskgroup: CREATE TEMPORARY TABLESPACE TEMP TEMPFILE '+TEMPDG' SIZE 10240M AUTOEXTEND ON; 5. Assign the default TEMP tablespace for the database back to the TEMP tablespace: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp; 6. Drop the TEMP2 tablespace: DROP TABLESPACE temp2 INCLUDING CONTENTS AND DATAFILES; The TEMP tablespace now resides on the IO Accelerator in the +TEMPDG disk group. Hosting redo logs on the IO Accelerator Because of the IO Accelerator low-latency capabilities, moving the logs off of traditional storage to the IO Accelerator yields a significant performance benefit for write-latency-bound applications. Common workloads, system configurations, and applications that might be improved include: High-volume OLTP applications Demanding internet applications that are write-intensive Best practices for this architecture include using multiple IO Accelerators in a mirrored configuration for storing the data or replicating logs off-system. Example C: Migrating redo logs to an IO Accelerator on a local filesystem For this example: The database is a single-instance standalone database. All of the current database datafiles reside in the /u01/oradata/testdb directory. The database does not use OMFs. The current REDO log sets are not multiplexed. /u02/oradata is an appropriately configured and redundant IO Accelerator-backed filesystem. 1. Change the ownership and permissions for the /u02/oradata directory: chown -R oracle.oinstall /u02/oradata chmod 775 /u02/oradata 2. Log on as the oracle user: su - oracle Single-instance performance architectures 18

19 3. Create the subdirectory for the TESTDB database in the /u02/oradata directory: mkdir -p /u02/oradata/testdb The IO Accelerator is now configured as a filesystem and ready for use by the Oracle REDO log storage. 4. As the oracle user, move the REDO log members to the IO Accelerator. 5. Log on to sqlplus as sysdba: export ORACLE_SID=TESTDB sqlplus / as sysdba The datafiles that make up the existing REDO logs cannot be renamed/relocated while the database is open. Instead, new REDO logs are created in the new location, and then the old REDO logs are dropped. 6. Enter the following SQL query to display the current REDO log member information: Select a.group# "Group Nbr", b.thread# "Thread Nbr", substr(a.member,1,65) "Member", b.bytes/ size_kb "Size (MB)", b.sequence# "Seq Nbr", b.archived "Archived", b.status "Status" from v$logfile a, v$log b order by b.thread#, a.group#, substr(a.member,1,65); To relocate the REDO logs, you must create new REDO log groups using the new IO Accelerator location, and then drop the old REDO log groups. 1. Use the following PL/SQL script to migrate the online REDO log groups to the new IO Accelerator filesystem location. For each online REDO log group, the script adds a log file in the new location, archives the current redo logs, and then drops the old log file. declare cursor rlc is select group# grp, thread# thr, bytes/1024 bytes_k, 'NO' srl from v$log union select group# grp, thread# thr, bytes/1024 bytes_k, 'YES' srl from v$standby_log order by 1; stmt varchar2(2048); swtstmt varchar2(1024) := 'alter system switch logfile'; ckpstmt varchar2(1024) := 'alter system checkpoint global'; begin for rlcrec in rlc loop if (rlcrec.srl = 'YES') then stmt := 'alter database add standby logfile thread ' rlcrec.thr ' ''/u02/oradata/testdb/stdby_redo0' rlcrec.grp '.log'' size ' rlcrec.bytes_k 'K'; execute immediate stmt; stmt := 'alter database drop standby logfile group ' rlcrec.grp; Single-instance performance architectures 19

20 execute immediate stmt; else stmt := 'alter database add logfile thread ' rlcrec.thr ' ''/u02/oradata/testdb/redo0' rlcrec.grp '.log'' size ' rlcrec.bytes_k 'K'; execute immediate stmt; begin stmt := 'alter database drop logfile group ' rlcrec.grp; dbms_output.put_line(stmt); execute immediate stmt; exception when others then execute immediate swtstmt; execute immediate ckpstmt; execute immediate stmt; end; end if; end loop; end; / When the script completes, the database REDO logs reside in the IO Accelerator location. Because OMFs are not used, you must remove the old REDO log datafiles from the /u01/oradata/testdb directory using operating system commands. 2. Enter the following SQL query to display the new location of the REDO logs: select a.group# "Group Nbr", b.thread# "Thread Nbr", substr(a.member,1,65) "Member", b.bytes/ size_kb "Size (MB)", b.sequence# "Seq Nbr", b.archived "Archived", b.status "Status" from v$logfile a, v$log b order by b.thread#, a.group#, substr(a.member,1,65); Example D: Migrating redo logs to an IO Accelerator configured with ASM For this example: The database is a single-instance standalone database using ASM for datafile storage. All of the current database datafiles reside in the ASM disk group +DATADG. The DB_CREATE_FILE_DEST parameter for the TESTDB database is +DATADG. The ASM disk group +REDODG is built from IO Accelerators and configured with the appropriate level of redundancy. Single-instance performance architectures 20

21 1. Log on the TESTDB database as sysdba: export ORACLE_SID=TESTDB sqlplus / as sysdba NOTE: The datafiles that make up the existing REDO logs cannot be renamed or relocated while the database is open. Instead, you must create new REDO logs in the new location, and then drop the old REDO logs. To relocate the REDO logs, create new REDO log groups using the new IO Accelerator diskgroup location, and then drop the old REDO log groups. 2. Use the following PL/SQL script to migrate the online redo log groups to the new IO Accelerator diskgroup. For each online redo log group, the script adds a log file in the new location, archives the current redo logs, and then drops the old log file. declare cursor rlc is select group# grp, thread# thr, bytes/1024 bytes_k, 'NO' srl from v$log union select group# grp, thread# thr, bytes/1024 bytes_k, 'YES' srl from v$standby_log order by 1; stmt varchar2(2048); swtstmt varchar2(1024) := 'alter system switch logfile'; ckpstmt varchar2(1024) := 'alter system checkpoint global'; begin for rlcrec in rlc loop if (rlcrec.srl = 'YES') then stmt := 'alter database add standby logfile thread ' rlcrec.thr ' ''+REDODG'' size ' rlcrec.bytes_k 'K'; execute immediate stmt; stmt := 'alter database drop standby logfile group ' rlcrec.grp; execute immediate stmt; else stmt := 'alter database add logfile thread ' rlcrec.thr ' ''+REDODG'' size ' rlcrec.bytes_k 'K'; execute immediate stmt; begin stmt := 'alter database drop logfile group ' rlcrec.grp; dbms_output.put_line(stmt); execute immediate stmt; exception when others then execute immediate swtstmt; execute immediate ckpstmt; execute immediate stmt; end; Single-instance performance architectures 21

22 end if; end loop; end; / When the script completes, the database REDO logs reside in the IO Accelerator location. Because ASM is used, the old REDO log datafiles are automatically deleted when the corresponding REDO log group is dropped. 3. Use the following SQL query to display the new location of the REDO logs: select a.group# "Group Nbr", b.thread# "Thread Nbr", substr(a.member,1,65) "Member", b.bytes/ size_kb "Size (MB)", b.sequence# "Seq Nbr", b.archived "Archived", b.status "Status" from v$logfile a, v$log b order by b.thread#, a.group#, substr(a.member,1,65); Using the IO Accelerator with flash cache Using the IO Accelerator as a flash cache target can yield significant performance benefits for read-heavy or query-heavy workloads where not all frequently read data can fit in RAM cache. The flash cache target enables the IO Accelerator to act as a warm data cache for reads. Using a flash cache might significantly reduce the load imposed on other storage devices, which could also yield improved write performance. See the Oracle website ( for additional information on configuring flash cache with Oracle databases. Prerequisites To use flash cache, you must be running a supported version of Linux or Solaris. The flash cache option is available only in Oracle Database 11gR2 and later. Assessing flash cache performance improvements To identify whether your current workload can benefit from adding flash cache, verify that the following conditions are true: The buffer pool advisory section of either the AWR report or the STATSPACK report indicate that a larger buffer cache can be beneficial. The db file sequential read is a top wait event. The CPUs have idle cycles. While running a heavy workload, you can issue the top command. If CPU cycles are in the idle or io wait categories, then those are unused CPU cycles. Single-instance performance architectures 22

23 Setting up flash cache Database Smart Flash Cache is an optional memory component in Oracle 11g Release 2 that you can add if your database is running on Solaris with Exadata storage or on Oracle Enterprise Linux. It is an extension of the SGA-resident buffer cache, providing a level 2 cache to the SGA, level 1 for database blocks. It can improve response time and overall throughput. The flash cache is an extension of the database buffer cache that is stored on a flash disk, a storage device that uses flash memory. Only data buffered in the SGA can make its way into the Database Flash Cache. The data is essentially aged out and written by DBWR into the flash cache device file. A flash cache is an extension of the database buffer cache that lives on a flash disk, which is a solid state storage device that uses flash memory. Without flash cache, the database reuses each clean buffer in main memory as needed, overwriting it. If the overwritten buffer is needed later, then the database must read it from magnetic disk. With flash cache, the database can write the body of a clean buffer to the flash cache, enabling reuse of its main memory buffer. The database keeps the buffer header in an LRU list in main memory to track the state and location of the buffer body in the flash cache. If this buffer is needed later, then the database can read it from the flash cache instead of from magnetic disk. In an Oracle RAC environment, you must configure flash cache on either all or none of the database instance nodes. Enabling flash cache Enabling the flash cache option requires setting two new flash cache parameters: DB_FLASH_CACHE_FILE and DB_FLASH_CACHE_SIZE. On the OEL platform, Metalink patch must be installed to enable flash cache. If you try to enable the flash cache option on a non-supported platform, you receive the following error message: ORA-00439: feature not enabled: Server Flash Cache ORA-01078: failure in processing system parameters The DB_FLASH_CACHE_FILE parameter specifies the path and file name for the file to contain flash cache, in either the operating system file system (such as /dev/fioa) or an Oracle ASM disk group (such as +FCDG). If the file does not exist, the database creates it during startup. The file must reside on a flash disk. If you configure flash cache on a disk drive (spindle), the database might not open or performance might suffer. The DB_FLASH_CACHE_SIZE parameter specifies the size of the flash cache. This value must be less than or equal to the physical memory size of the flash disk device. This parameter can only be specified at instance startup expressed as ng, indicating the number of gigabytes. You can dynamically change this parameter to 0 (disabling the flash cache) after the database is started. You can re-enable flash cache by setting this parameter to the original value when the database was started. Dynamic resizing of DB_FLASH_CACHE_SIZE or re-enabling flash cache to a different size is not supported. Sizing the flash cache As a general rule, size the flash cache to be between two times and 10 times the size of the buffer cache. Any multiplier less than two is not beneficial. If you are using automatic shared memory management, Single-instance performance architectures 23

24 make the flash cache between two times and 10 times the size of SGA_TARGET. Using 80% of the size of SGA_TARGET instead of the full size is sufficient. Tuning memory for the flash cache For each database block moved from the buffer cache to the flash cache, a small amount of metadata about the block is kept in the buffer cache. For a single instance database, the metadata consumes approximately 100 bytes. For an Oracle RAC database, it is closer to 200 bytes. You must therefore take this extra memory requirement into account when adding the flash cache. If you are managing memory manually, increase the size of the buffer cache by an amount approximately equal to the number of database blocks that fit into the flash cache multiplied by 100 (or 200 for Oracle RAC). If you are using automatic memory management, increase the size of MEMORY_TARGET using the algorithm previously described. You might have to increase the size of MEMORY_MAX_TARGET. If you are using automatic shared memory management, increase the size of SGA_TARGET. You can choose to not increase the buffer cache size to account for the flash cache. In this case, the effective size of the buffer cache is reduced. However, you can offset this loss by using a larger flash cache. Using flash cache When the flash cache option has been enabled, schema Table objects can specify the storage clause option FLASH_CACHE in the CREATE TABLE statement. The FLASH_CACHE clause lets you override the automatic buffer cache policy and specify how specific schema objects are cached in flash memory. To use this clause, Database Smart Flash Cache (flash cache) must be configured on your system. Because flash memory is faster than magnetic disks, the database can improve performance by caching buffers in the flash cache instead of reading from magnetic disk. The values for the FLASH_CACHE clause are KEEP, NONE, and DEFAULT. KEEP Specify KEEP if you want the schema object buffers to remain cached in the flash cache as long as the flash cache is large enough. NONE Specify NONE to ensure that the schema object buffers are not cached in the flash cache. This enables you to reserve the flash cache space for more frequently accessed objects. DEFAULT Specify DEFAULT if you want the schema object buffers to be written to the flash cache when they are aged out of main memory, and then be aged out of the flash cache with the standard buffer cache replacement algorithm. This is the default if flash cache is configured and you do not specify KEEP or NONE. Example: Flash cache use To enable and use flash cache with Oracle 11gR2 on OEL 5 64-bit: 1. Download and install Metalink Patch Patch to enable Oracle Database Smart Flash Cache for OEL. 2. Set the following Oracle initialization parameters: db_flash_cache_file='/dev/fioa1' Single-instance performance architectures 24

25 db_flash_cache_size=xxg (this value depends on the size of the SSD device) 3. Stop and then restart the Oracle database. To use flash cache, create or update schema objects with the flash_cache storage clause specified. For example: SQL> create table TB_TEST01 storage( flash_cache keep) as select * from all_objects; Table created. SQL> select table_name, flash_cache from user_tables; TABLE_NAME FLASH_C TB_TEST01 KEEP For more information on the options for the flash_cache storage clause, see "Using flash cache (on page 24)." Single-instance performance architectures 25

26 Oracle RAC performance architectures SRP and iser targets SRP and iser are RDMA-capable block protocols that can be sent over RDMA-capable networks such as Infiniband. IO Accelerators are a good match for use with SRP/iSER because of the high performance they can deliver over the wire. By setting up two or three separate SRP or iser target servers and using ASMs built in replication capabilities (levels NORMAL or HIGH redundancy), a fault-tolerant, high performance RAC cluster can be built. Oracle RAC performance architectures 26

27 Acronyms and abbreviations ADDM automatic database diagnostic monitor ASM Advanced Server Management AWR automatic workload repository CPU central processing unit DBWR database writer EM Enterprise Manager HTML hypertext markup language I/O input/output iscsi Internet Small Computer System Interface iser iscsi extensions for RDMA LGWR log writer LRU least recently used Acronyms and abbreviations 27

28 LVM Logical Volume Manager OEL Oracle Enterprise Linux OLTP online transaction processing OMF Oracle Managed File RAC real application clusters RAID redundant array of inexpensive (or independent) disks RAM random access memory RDMA Remote Direct Memory Access SCSI small computer system interface SGA system global area SRP SCSI remote protocol Acronyms and abbreviations 28