Oracle Database 10g. Several architectural enhancements have been introduced. Best Practices for. Automatic Storage Management on Dell/EMC Storage

Transcription

1 Best Practices for Oracle Database 10g Automatic Storage Management on Dell/EMC Storage A highly available and scalable storage system forms the heart of data centers running Oracle databases This article reflects the cumulative storage design and tuning recommendations from Dell and Oracle teams on Oracle Database 10g solutions using Dell PowerEdge servers, Dell/EMC Fibre Channel storage, and Oracle Automatic Storage Management on the Red Hat Linux operating system BY PAUL RAD, RAMESH RAJAGOPALAN, TESFAMARIAM MICHAEL, AND JAY KOZAK Several architectural enhancements have been introduced in Oracle Database 10g that can benefit administrators tasked with deploying and managing an Oracle database solution The enhanced techniques and methods supported by Oracle Database 10g either automate or greatly simplify the process of configuring, monitoring, and managing an Oracle database, and can assist organizations with implementing the degree of availability and performance that best meets their service-level objectives Key techniques include: Automatic Storage Management (ASM): For optimal layout of database files Automatic Workload Repository (AWR): For gathering performance data Automatic Database Diagnostics Monitor (ADDM): For analyzing performance issues Automatic Workload Management (AWM): For managing and controlling processing resources required for applications Virtualization and provisioning: For efficient, on-demand usage of processing resources ASM addresses one of the major challenges faced by database administrators: the storage management process, which involves creating and tuning a storage layout for database files, identifying hot spots, monitoring I/O distribution among the physical disks, and monitoring overall storage capacity (see Figure 1) As the database grows in size, this often-daunting storage management process needs to be repeated Moreover, many of the associated administrative tasks require taking the database offline, which can reduce availability to a level that may not be acceptable ASM can help to improve the storage management process and to reduce total cost of ownership (TCO) by allowing storage to grow as needed without requiring up-front investments that account for future growth Before ASM, raw devices and the Oracle Cluster File System (OCFS) were the available storage management options for an Oracle database running on the Linux operating system However, these methods are subject to the following limitations: Data file management on raw devices can be difficult with respect to name space mapping and backup schemes wwwdellcom/powersolutions Reprinted from Dell Power Solutions, October 2004 Copyright 2004 Dell Inc All rights reserved POWER SOLUTIONS 91

2 Create and tune storage layout Monitor disk I/O distribution and storage capacity Configuring the private network for more information) Primary communication among the cluster nodes takes place over the private network Clients and other application servers access the database over the public network Figure 1 Managing storage for a database Identify hot spots Storage must be allocated for both raw devices and OCFS with future growth in mind resulting in significant initial costs A storage solution is not scalable if storage allocation is based only on short-term requirements Expanding the storage for a data file requires creating a larger data file, exporting the data from the existing data file, and importing the data into the larger data file During these operations, the database is not available Both raw devices and OCFS abstract the organization (RAID level) of the physical disk group Consequently, an application such as a database server cannot take advantage of physical properties host bus adapters (s), SCSI adapters, bus speeds, disk spindles, and so forth to balance I/O or redistribute data as the data file grows Dell/EMC storage systems The modular Dell/EMC CX series of Fibre Channel storage systems incorporates RAID, multipath I/O, and on-demand storage expansion capabilities to help provide cost-effective, continuous availability for critical business environments through either SAN or direct attach configurations (see Figure 3 for representative entry-level, midrange, and enterprise-class models) A SAN architecture allows administrators to increase storage incrementally and add servers as business demands grow Key features of Dell/EMC CX series storage systems include the following: High availability: Modular, redundant hardware architecture, combined with switches, creates multiple paths to the storage to help provide business continuance Scalability and capacity: Up to 35 TB of storage can be supported using Fibre Channel 2 (FC2) drives Disks and enclosures can be added as needed, resulting in efficient utilization of storage Manageability: EMC Navisphere and VisualSAN software help provide simple, powerful storage management capabilities, including nondisruptive software upgrades This article provides an overview of Oracle Real Application Clusters (RAC) 10g running over a storage area network (SAN) that comprises Dell servers and Dell/EMC storage It also describes how Dell/EMC storage systems and ASM can help improve database performance and availability Public network Dedicated network Private network (bonded NICs) Private network (bonded NICs) Implementing Oracle RAC 10g on Dell clusters Dell and Oracle have developed a RAC configuration that is based on a SAN comprising up to eight Dell PowerEdge server nodes, a Dell/EMC CX series Fibre Channel storage enclosure, a Fibre Channel network, a private network, and a public network (see Figure 2) The storage enclosure can be connected to the nodes directly or through Fibre Channel switches The PowerEdge server nodes run the Red Hat Enterprise Linux AS 3 operating system with Update 2 or higher and Oracle Database 10g Enterprise Edition database software 1 An Oracle RAC 10g cluster requires a private network and a public network The private network uses two Gigabit 2 Ethernet network interface cards (NICs) that are bonded together using the Red Hat Enterprise Linux network bonding feature (see the sidebar Fibre Channel switch Public network Fibre Channel network Private network Dell PowerEdge servers Dell/EMC Fibre Channel storage Figure 2 Architecture of Oracle RAC 10g cluster with Dell/EMC Fibre Channel storage Fibre Channel switch 1 To review the latest Dell and Oracle supported configurations on Dell PowerEdge servers, visit 2 This term does not connote an actual operating speed of 1 Gbps For high-speed transmission, connection to a Gigabit Ethernet server and network infrastructure is required 92 POWER SOLUTIONS October 2004

3 Dell/EMC Maximum Maximum storage number storage capacity system of disks with FC2 drives Specifications CX in 1 disk processor 88 TB Entry-level array enclosure (DPE) and Bandwidth: 680 MB/sec; 3 disk array enclosures 50,000 I/Os per second (IOPS) (DAEs) RAID levels: 0, 1, 3, 5, and 10 2 GB cache 2 Gbps DPE2 Multipath I/O CX in 1 DPE 175 TB Midrange array and 7 DAEs Bandwidth: 780 MB/sec; 120,000 IOPS RAID levels: 0, 1, 3, 5, and 10 4 GB cache 2 Gbps DPE2 Multipath I/O CX in 16 DAEs 35 TB Enterprise-class array Bandwidth: 1520 MB/sec; 200,000 IOPS Figure 3 Specifications for Dell/EMC storage system enclosures RAID levels: 0, 1, 3, 5, and 10 8 GB cache Storage processor enclosure (SPE) Multipath I/O RAID technology The Dell/EMC CX series storage systems use RAID technology RAID technology groups separate inexpensive disks into one logical unit number (LUN) to help improve reliability, performance, or both This approach spreads data across all disks, which are partitioned into units called stripe elements Depending on the RAID level, the storage-system hardware can read from and write to multiple disks simultaneously and independently Because this approach enables several read/write heads to work on the same task at once, RAID can enhance performance The chunk data size read from or written to each disk at a time makes up the stripe element size Figure 4 shows an example six-disk RAID-10 configuration in which each primary disk is first striped and then mirrored to another disk To set up network bonding for Broadcom or Intel NICs and to configure the private network, perform the following steps on each cluster node: 1 Log in as root CONFIGURING THE PRIVATE NETWORK 2 Add the following line to the /etc/modulesconf file: alias bond0 bonding 3 For high availability, edit the /etc/modulesconf file and set the option for link monitoring The default value for miimon is 0, which disables link monitoring Change the value to 100 milliseconds initially, and adjust it as needed to improve performance: options bonding miimon=100 4 In the /etc/sysconfig/network-scripts/ directory, edit the ifcfg-bondn configuration file for bond number n For example, the configuration file ifcfg-bond0 for the first bond (bond0) would appear as follows: DEVICE=bond0 IPADDR= ONBOOT=yes BOOTPROTO=none USERCTL=no 5 To use the virtual bonding device (such as bond0), all members of bond0 must be configured so that MASTER=bond0 and SLAVE=yes For each member of the specified bonding device, edit its respective configuration file ifcfg-ethn in /etc/sysconfig/ network-scripts/ as follows: DEVICE=ethn HWADDR=MAC ADDRESS ONBOOT=yes Disk 1 primary Disk 1 mirror TYPE=Ethernet USERCTL=no MASTER=bond0 Stripe size Disk 2 primary Disk 2 mirror SLAVE=yes BOOTPROTO=none Disk 3 primary Enter the following command: Disk 3 mirror service network restart Figure 4 RAID-10 with six disks wwwdellcom/powersolutions POWER SOLUTIONS 93

4 Multipath I/O The Dell/EMC CX series storage systems are designed to support multipath routing between SAN switches 3 In a typical topology, a node has multiple Fibre Channel s, each of which is connected to the ASM enables database same SAN, resulting in multiple paths to the same storage system administrators to manage devices Redundant paths in a SAN help provide failover capability when any component in the a small number of disk data path fails Multiple paths can groups and automatically also enhance efficiency, allowing administrators to load balance SAN traffic by distributing manages the placement I/O across all available paths of data files within This approach can enable the SAN to take advantage of the those disk groups additional bandwidth provided by each physical connection EMC PowerPath software is designed to work with Dell/EMC storage systems to help create efficient I/O path management PowerPath also helps enhance the high-availability capabilities of Dell/EMC storage systems because it is designed to automatically detect and recover from server-to-storage path failures ASM ASM is an Oracle database specific file system and volume manager that is built into the Oracle Database 10g kernel 4 ASM is designed to allow Oracle Database 10g to directly manage raw disks and help eliminate the need for a file system and volume manager to manage Oracle files including data files, log files, archive logs, control files, and recovery manager backup sets Database administrators can manage the creation of storage pools by using new SQL commands, Oracle Enterprise Manager 10g, or Oracle Database Creation Assistant ASM virtualizes storage into disk groups, which serve as repositories for Oracle database files A disk group is a logical grouping of several individual disks in a storage array or several RAID LUNs ASM enables database administrators to manage a small number of disk groups and automatically manages the placement of data files within those disk groups ASM helps provide the following features and functions, which can be critical for an enterprise database environment: Dynamic provisioning: Dynamically scales storage capacity without affecting database availability I/O balancing: Automatically balances I/O across all available disks defined in the storage environment Also helps prevent hot spots and maximize performance by adjusting rapidly to changing data-access patterns High availability: Provides options for fault tolerance and high availability by creating data redundancy through software when hardware RAID is not available In ASM, software redundancy is called internal redundancy, whereas redundancy created through hardware RAID is known as external redundancy Setting up ASM on Dell/EMC storage Oracle has many file types such as data, log, temp, archive, undo, system, control, and backup each characterized by different I/O behavior In addition, the Oracle I/O workload is application dependent Understanding the I/O workload for each database file used by an application can be difficult, and a new or unfamiliar application can make this task daunting This section provides an overview of various ASM techniques that can be used to configure a Dell/EMC CX series storage system for optimal performance with Oracle Database 10g Implementing hardware mirroring Through hardware mirroring, ASM can be configured with external redundancy, which typically performs better than internal redundancy because the server Because ASM offers the does not need to consume CPU cycles for managing redundancy capability to implement Instead, these cycles are available for the database or for other host-based striping, using operating system related operations The simplest way to help ASM with storage-system ensure that data is not lost is to implement mirroring at the RAID striping such as storage subsystem level Generally, the most straightforward RAID-10 can generate an approach is to implement hardware mirroring at the disk or partition level and then implement optimal, double-striped striping on top of the mirrored (a stripe on a stripe) layout disks Mirrored data can be lost only when multiple disk failures for spreading I/O across occur and the probability that multiple disk drive failures will occur at the same time is relatively multiple disk spindles 3 For more information, see Building a Highly Scalable and Available Data Environment for Oracle9i RAC by Zafar Mahmood, Paul Rad, and Robert Nadon in Dell Power Solutions, November For more information about ASM, see Enabling a Highly Scalable and Available Storage Environment Using Oracle Automatic Storage Management by Zafar Mahmood, Joel Borellis, Mahmoud Ahmadian, and Paul Rad in Dell Power Solutions, June POWER SOLUTIONS October 2004

5 USING ASM TO CREATE A DISK GROUP This sidebar describes how to set up shared storage on Dell/EMC CX series storage systems using ASM to create a disk group with nine raw partitions: 1 Log in as root 2 Enter the following commands to change the names of the raw devices: mv /dev/raw/raw1 /dev/raw/asm1 mv /dev/raw/raw2 /dev/raw/asm2 mv /dev/raw/raw8 /dev/raw/asm8 mv /dev/raw/raw9 /dev/raw/asm9 3 Enter the following commands to set ownership of the devices to user oracle of group dba: chown oracledba /dev/raw/asm1 chown oracledba /dev/raw/asm2 chown oracledba /dev/raw/asm8 chown oracledba /dev/raw/asm9 4 Edit the /etc/sysconfig/rawdevices file and add the following lines: /dev/raw/asm1 /dev/emcpowerb /dev/raw/asm2 /dev/emcpowerc /dev/raw/asm8 /dev/emcpoweri /dev/raw/asm9 /dev/emcpowerj 5 Enter the command: service rawdevices restart 6 Create the initasmora file containing the following line: INSTANCE_TYPE = +ASM 7 Connect to the Oracle SQL*Plus command-line interface as SYSDBA for SID=ASM 8 Enter the following commands: STARTUP NOMOUNT PFILE=initASMora CREATE SPFILE FROM PFILE=initASMora CREATE DISKGROUP dgroup1 EXTERNAL REDUNDANCY DISK '/dev/raw/asm1','/dev/raw/asm2','/dev/raw/asm3', '/dev/raw/asm4','/dev/raw/asm5','/dev/raw/asm6', '/dev/raw/asm7','/dev/raw/asm8','/dev/raw/asm9' 9 Create the initasmdbora file containing the lines: INSTANCE_TYPE = RDBMS DB_CREATE_FILE_DEST = '+dgasm' 10 Connect to SQL*Plus as SYSDBA for SID=ASMDB 11 Enter the following command: STARTUP NOMOUNT PFILE=initASMDBora small given the highly reliable nature of current disk drives Also, many systems allow a spare drive to be configured so that repairs can be performed automatically after a RAID drive failure Using a hot spare A required component of an enterprise storage setup, the hot spare is a dedicated replacement disk on which end-user applications cannot store information Instead, the hot spare is reserved as a global replacement: if any disk in a RAID-5 group, RAID-1 mirrored pair, or RAID-10 group fails, the storage processor (SP) automatically rebuilds the failed disk s structure on the hot spare When the SP finishes rebuilding the hot spare, the disk group functions as usual, using the hot spare instead of the failed disk When the failed disk is replaced, the SP copies the data from the hot spare onto the replacement disk Establishing a double-striped layout Striping is a technique available on Dell/EMC storage systems that can enhance performance by spreading the I/O load across multiple disk spindles Striping all files across all disks can help ensure that the full bandwidth of all disk drives is available for any operation To optimize disk bandwidth and use all available disks, the data to be accessed must be spread across as many disks as possible so that every disk in the disk farm has roughly equivalent utilization Any disk that is used more than the other disks could become a performance bottleneck Striping all files across all disks can help wwwdellcom/powersolutions POWER SOLUTIONS 95

6 equalize the load across disk drives and help eliminate hot spots This process is also designed to improve response time by shortening disk queues Because ASM offers the capability to implement host-based striping, using ASM with storage-system RAID striping such as RAID-10 can generate an optimal, double-striped (a stripe on a The simplest way to stripe) layout for spreading I/O across multiple disk spindles help ensure that data is Figure 5 shows a sample configuration in which each LUN in not lost is to implement the storage array with multiple disk array enclosures (/dev/raw/ mirroring at the storage ASM1, /dev/raw/asm2, and so on) is configured as RAID-10 subsystem level When the LUNs are specified to ASM as physical devices, ASM can perform another level of striping among the LUNs to generate a double-striped layout The sidebar Using ASM to create a disk group provides more information about setting up the nine partitions shown in Figure 5 Simplifying the management of database files Kept and maintained by the Oracle kernel, ASM is a file system created specifically for Oracle data files on top of raw devices ASM is designed to distribute I/O on top of ASM disk groups, which comprise one or more raw disks that are mirrored by the storage system The disk group is used for creating files, and 1 MB files can be extended across the available disks The more spindles that are used within a disk group, the more flexibility the Oracle Database 10g server has to spread the I/O among disks, which can help increase performance and improve data redundancy ASM is designed to help simplify file management in an Oracle database By using this Oracle Database 10g feature, database administrators can easily create a highly scalable storage solution from a set of disks ASM provides the necessary tool set to free database administrators from the mundane tasks of adding, shifting, and removing disks helping to improve availability and response time while contributing to reduced TCO Paul Rad is a senior software engineer in the Dell Database and Application Engineering Department of the Dell Product Group Paul has master s degrees in both Computer Science and Computer Engineering from The University of Texas at San Antonio Ramesh Rajagopalan is a lead software engineer in the Dell Database and Application Engineering Department of the Dell Product Group His current areas of focus include Oracle RAC solutions and performance analysis of Dell cluster solutions Ramesh has a Bachelor of Engineering in Computer Science from the Indian Institute of Science, Bangalore Node 1 Oracle Database 10g ASM PowerPath Private NIC Node 2 Oracle Database 10g ASM PowerPath Request Request Request Request ASM disk group using nine LUNs (data files) ASM disk group using nine LUNs (flashback recovery) Tesfamariam Michael is a software engineer in the Dell Database and Application Engineering Department of the Dell Product Group Tesfamariam has an MS in Computer Science and a BS in Mathematics from Clark Atlanta University, and a BS in Electrical Engineering from Georgia Institute of Technology Jay Kozak has been with Oracle since 1996 and is currently a technical business development manager in the Server Technologies organization Recently, Jay has been primarily focused on the joint Oracle and Dell engineering initiatives Jay graduated from Northern Illinois University with a BS in Finance in 1989 and has done post-graduate computer science work at the Illinois Institute of Technology Fibre Channel switches /dev/raw/asm1 /dev/raw/asm2 /dev/raw/asm3 /dev/raw/asm4 /dev/raw/asm5 /dev/raw/asm6 /dev/raw/asm7 /dev/raw/asm8 /dev/raw/asm9 Dell/EMC CX700 Figure 5 Using ASM to implement double striping on a Dell/EMC CX700 with multiple disk array enclosures configured as RAID-10 FOR MORE INFORMATION Oracle Automatic Storage Management: Dell and Oracle supported configurations: Dell/EMC storage: 96 POWER SOLUTIONS October 2004