EMC SCALING PERFORMANCE FOR ORACLE VIRTUAL MACHINE

Transcription

1 White Paper EMC SCALING PERFORMANCE FOR ORACLE VIRTUAL MACHINE Automate performance Scale OLTP workloads Rapid provisioning of Oracle databases EMC Solutions Group Abstract This solution illustrates the benefits of deploying EMC FAST Suite for Oracle OLTP databases in an optimized, scalable virtual environment. The Oracle Real Application Cluster (RAC) 11g database accesses an EMC VNX 7500 storage using the Oracle direct NFS (dnfs) Client enabling simplified configuration, improved performance, and enhanced availability. EMC s SnapSure technology and the Oracle dnfs clonedb feature create rapid provisioning of Oracle databases. Oracle VM provides the virtualization platform. September 2012

2 Copyright 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H10739 EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 2

3 Table of contents Executive summary... 6 Business case... 6 Solution overview... 6 Key results... 7 Introduction... 8 Purpose... 8 Scope... 8 Audience... 8 Terminology... 8 Technology overview Introduction EMC VNX EMC FAST Suite (FAST VP, FAST Cache) EMC FAST VP EMC FAST Cache EMC SnapSure Oracle VM Oracle RAC Oracle dnfs Solution architecture Introduction Solution architecture Hardware resources Software resources Oracle storage layout Oracle file system allocation on VNX Oracle dnfs Client configuration Configuring Oracle databases Database and workload profile Oracle database schema Configuring FAST Cache on EMC VNX Overview Analyze the application workload Create and enable/disable FAST Cache FAST Cache best practices for Oracle

4 Configuring FAST VP on EMC VNX Overview Tiering policies Configure FAST VP Configuring Oracle virtual machines Overview Step 1: Prepare Oracle VM Server Step 2: Install Oracle VM Manager Step3: Configure server pool Step 4: Configure network Step 5: Configure Oracle virtual machines Node scalability test Test objective Test procedure Test result FAST Suite test FAST Suite and manual tiering comparison FAST Cache test FAST Cache warm-up FAST Cache test procedure FAST VP test FAST VP moving data across tiers FAST VP test procedure FAST Suite test FAST Suite test procedure Test result FAST Suite effects on database transactions per minute FAST Suite performance on read response time Wait statistics from Oracle AWR reports Statistics from Unisphere for VNX dnfs clonedb test Test objectives Test procedures Test results Resilience test Test objectives Test procedures EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 4

5 Physical NIC failure Data mover panic Test result Physical NIC failure Data mover panic Conclusion Summary Findings References White papers Product documentation Other documentation

6 Executive summary Business case Oracle databases and their dependant applications are mission-critical for businesses. They require high performance with low I/O latency. As a result, Oracle environments must address an increasingly broad range of business demands, including the ability to: Scale Oracle online transaction processing (OLTP) workloads for performance. Maximize performance while reducing the cost of ownership of the system. Traditional methods of using Flash drive technology are manual, complex, repetitive, and disruptive EMC FAST Suite automatically and nondisruptively tunes an application, based on the access patterns. FAST Cache services active data with less Flash drives, while FAST VP optimizes disk utilization and efficiency with Serial Attached SCSI (SAS) and Near-Line SAS (NL-SAS) drives. Meet on-demand Oracle requirements to create, deploy, and manage numerous production, development, and testing environments. This solution addresses all these challenges for a scalable virtualized Oracle RAC 11g database deployment. Solution overview This solution uses the following technologies to support the demands of the growing enterprise infrastructure: EMC VNX 7500 series EMC Unisphere EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP) EMC FAST Cache EMC SnapSure checkpoint Oracle VM Oracle direct NFS (dnfs) Oracle dnfs clonedb Technologies such as simplified storage management and fully automated storage tiering (FAST ) provide an infrastructure foundation that meets the following business needs: Efficient Automate Oracle performance tuning With FAST VP and FAST Cache enabled, the storage array continuously tunes an application, based on the access patterns. Cost saving Improve the total cost of ownership (TCO) EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 6

7 FAST Cache can manage active data with fewer Flash drives, while FAST VP optimizes disk utilization and efficiency across SAS and NL-SAS drives. Scale Support growing Oracle workloads that requires increasing high I/Os per second (IOPS) enabled by scaling out a virtual Oracle Real Application Clusters (RAC) node with Oracle dnfs Client and the latest 10 GbE data center technology. Agile Rapid cloning of Oracle environments such as testing, development, and patch databases by using Oracle dnfs clonedb technology. Key results This solution demonstrates the following key results: Performance improvement with FAST Suite: 2 times improvement in transactions per minute (TPM) 3.5 times improvement in IOPS 30 times decrease in average read response times 91 percent hit ratio after a warm-up period of FAST Cache Simple management Only a few steps are required to configure FAST VP and FAST Cache. Customers can enable or disable FAST Cache and FAST VP without affecting the system. Nondisruptive FAST VP and FAST Cache can identify hot data automatically and nondisruptively. This frees Oracle database administrators (DBAs) from the complex, repetitive, and manual processes of tuning the back-end storage. Scale Customers can easily and nondisruptively scale out Oracle virtual RAC nodes as application needs evolve, enabling customers to take an incremental approach to address growing workload needs. Agile The EMC SnapSure checkpoint and the Oracle dnfs clonedb feature enable Oracle DBAs to rapidly deploy additional database copies from a production database for testing, development, or other purposes, while minimizing the storage capacity requirements for those additional database instances. Resilient The EMC VNX standby data mover and the Oracle dnfs Client enable high availability for Oracle RAC database. The database is still up during the physical NIC failure and the data mover panic, enabling a resilience database with automatic failover. 7

8 Introduction Purpose Scope Audience Terminology This white paper describes the design, testing, and validation of an Oracle OLTP system using the EMC VNX7500 storage platform, EMC FAST Suite, EMC SnapSure, Oracle VM, Oracle dnfs, and the Oracle dnfs clonedb technologies. This solution demonstrates the performance, scalability, and resiliency of the Oracle RAC database on EMC infrastructure. The scope of the white paper is to: Introduce the key solution technologies. Describe the solution architecture and design. Describe the solution scenarios and present the results of validation testing. Identify the key business benefits of the solution. This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators and virtualization administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments. This paper includes the following terminology. Table 1. Terminology Acronym AWR dnfs FAST VP FC IOPS LUN NIC NFS ODM OLTP OVM OVS RAC SAS Term Automatic Workload Repository Direct NFS Fully Automated Storage Tiering for Virtual Pools Fibre Channel I/Os per second Logical unit number Network interface card Network File System Oracle Disk Manager Online transaction processing Oracle virtual machine Oracle VM Server Real Application Clusters Serial Attached SCSI EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 8

9 Acronym SCSI SGA TCO TPM VNX OE Term Small Computer System Interface System global area Total cost of ownership Transactions per minute VNX Operating Environment 9

10 Technology overview Introduction EMC VNX7500 The solution uses the following hardware and software components: EMC VNX7500 EMC FAST Suite EMC SnapSure Oracle VM Oracle Database 11g Release 2 Enterprise Edition with Oracle Clusterware Oracle dnfs Client VNX7500 is a member of the VNX series next-generation storage platform, which is powered by Intel quad-core Xeon 5600 series processors and delivers five 9s availability. The VNX series is designed to deliver maximum performance and scalability for enterprises, enabling them to dramatically grow, share, and costeffectively manage multiprotocol file and block systems. The VNX Operating Environment (VNX OE) allows Microsoft Windows and Linux/UNIX clients to share files in multiprotocol NFS and Common Internet File System (CIFS) environments. VNX OE also supports iscsi, FC, and Fibre Channel over Ethernet (FCoE) access for high-bandwidth and latency-sensitive block applications. EMC FAST Suite (FAST VP, FAST Cache) The FAST Suite for VNX arrays includes FAST VP and FAST Cache. EMC FAST VP VNX FAST VP is a policy-based, auto-tiering solution for enterprise applications. FAST VP operates at a granularity of 1 GB, referred to as a "slice". The goal of FAST VP is to efficiently use storage tiers to lower a customer's TCO by tiering colder slices of data to high-capacity drives, such as NL-SAS, and to increase performance by keeping hotter slices of data on performance drives, such as Flash drives. This process occurs automatically and transparently to the host environment. High locality of data is important to realize the benefits of FAST VP. When FAST VP relocates data, it moves the entire slice to the new storage tier. To successfully identify and move the correct slices, FAST VP automatically collects and analyzes statistics before relocating data. Customers can initiate the relocation of slices manually or automatically by using a configurable, automated scheduler that can be accessed from the Unisphere management tool. The multitiered storage pool allows FAST VP to fully use all three storage tiers: Flash, SAS, and NL-SAS. EMC FAST Cache FAST Cache uses Enterprise Flash drives to add an extra layer of cache between the dynamic random access memory (DRAM) cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 10

11 most active data is served from high-performing Flash drives and can reside on this faster medium for as long as is needed. FAST Cache tracks data activity at a granularity of 64 KB and promotes hot data into FAST Cache by copying it from the hard disk drives (HDDs) to the Flash drives assigned to FAST Cache. Subsequent I/O access to that data is handled by the Flash drives and is serviced at Flash drive response times this ensures very low latency for the data. As data ages and becomes less active, it is flushed from FAST Cache to be replaced by more active data. A small number of Flash drives implemented as FAST Cache provides a greater performance increase than a large number of short-stroked HDDs. FAST Cache is particularly suited to applications that randomly access storage with high frequency, such as Oracle OLTP databases. In addition, OLTP databases have inherent locality of reference with varied I/O patterns. Applications with these characteristics benefit most from deploying FAST Cache. The optimal use of FAST Cache is achieved when the working dataset can fit within the FAST Cache. EMC SnapSure Oracle VM Oracle RAC Oracle dnfs SnapSure enables you to create point-in-time, logical images of a Production File System (PFS). SnapSure uses a "copy on first modify" principle. A PFS consists of blocks. When a block within the PFS is modified, a copy containing the block s original contents is saved to a separate volume called the SavVol. Subsequent changes made to the same block in the PFS are not copied into the SavVol. SnapSure reads the original blocks from the PFS in the SavVol and the unchanged PFS blocks remaining in the PFS according to a bitmap and blockmap data-tracking structure. These blocks combine to provide a complete point-in-time image called a checkpoint. Oracle VM is a server virtualization solution composed of Oracle VM Server for x86 and Oracle VM Manager. Oracle VM Server is composed of a hypervisor and privileged domain (Dom0) that allows multiple domains or virtual machines (such as Linux, Solaris, Windows, and so on.) to run on one physical machine. Oracle VM Manager controls the virtualization environment, and creates and monitors Oracle VM servers and the virtual machines. Oracle RAC extends Oracle Database so that you can store, update, and efficiently retrieve data using multiple database instances on different servers at the same time. Oracle RAC provides the software that manages multiple servers and instances as a single group. Oracle dnfs is an alternative to using kernel-managed NFS. With Oracle Database 11g release 2 (11.2), instead of using the operating system kernel NFS client, you can configure Oracle Database to access NFS V3 servers direct using an Oracle internal dnfs Client. This native capability enables direct I/O with the storage devices, bypassing the operating system file cache and reducing the need to copy data between the operating system and database memory. The dnfs Client also enables asynchronous I/O access to NFS appliances. 11

12 Solution architecture Introduction The solution is designed to test the following: Node scalability Performance of FAST Suite Provisioning test/development environments System resilience of an Oracle OLTP RAC database configured using dnfs We carried out the testing on an Oracle RAC 11g database using an EMC VNX7500 array as the underlying storage. Oracle VM was used as the virtualization platform. The VNX array was configured as an NFS server and the Oracle RAC nodes were configured to access the NFS server directly using the Oracle internal dnfs Client. Solution architecture Figure 1 depicts the architecture of the solution. With Oracle VM version installed, the Oracle VM server farm consists of two Oracle VM servers; four Oracle virtual machines (two on each VM server) were deployed as a four-node RAC database. As Oracle Support's suggestion, we deployed Oracle RAC for this virtualized solution. The storage and cluster interconnect networks used 10 Gigabit Ethernet (GbE). Figure 1. Architecture overview EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 12

13 Hardware resources Table 2 details the hardware resources for the solution. Table 2. Hardware resources Purpose Quantity Configuration Storage array 1 EMC VNX7500 with: 2 storage processors, each with24 GB cache 75 x 300 GB 10k 2.5 inch SAS drives 4 x 300 GB 15k 3.5 inch SAS drives (vault disk) 11 x 200 GB 3.5 inch Flash drives 4 x data movers( 2 primary and 2 standby) Dual-port 10 GbE for each data dover Oracle VM Server 2 4 x 8-core CPUs, 256 GB RAM, 2 x dual-port 1 Gb/s Ethernet NICs 2 x dual-port 10 Gb/s CNA NICs Ethernet switches 2 10 Gb/s Ethernet switches 2 1 Gb/s Ethernet switches Software resources Table 3 details the software resources for the solution. Table 3. Software resources Software Version Purpose EMC VNX OE for block VNX operating environment EMC VNX OE for file VNX operating environment Unisphere VNX management software Oracle Grid Infrastructure Oracle ASM, Oracle Clusterware, and Oracle Restart Oracle Database Oracle Database and Oracle RAC Oracle Enterprise Linux Database server OS Oracle VM Oracle Virtual Machine Swingbench 2.4 TPC-C like benchmark tool 13

14 Oracle storage layout The disk configuration uses four back-end 6 Gb SAS ports within the VNX7500 storage array. Figure 2 illustrates the disk layout of the environment. Figure 2. Disk layout Note A Cluster Ready Services (CRS) pool was deployed on the data vault disks due to low I/O activities. Figure 3 shows a logical representation of the layout of the file system used for the Oracle datafiles. We used four data movers in a 2+2 active/standby configuration. Two active data movers were used to access the file systems, which were distributed evenly across the four SAS ports. The back-end configuration was based on the I/O requirements. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 14

15 Figure 3. Datafile system logical view Unisphere provides a simple GUI to create and manage the file systems. Figure 4 shows the usage of each file system and its serving data mover. It is well balanced for the workload. Figure 4. The file system information panel in Unisphere 15

16 Oracle file system allocation on VNX7500 Table 4 details the Oracle file system storage allocation on VNX7500. All the storage pools were created on 300 GB 10k SAS drives. Table 4. Oracle file system allocation on VNX7500 File type Raid type No. of LUNs Disk volumes (dvols) Size Data Mover FRA files 4+1 RAID 5 10 D11 to D20 4 TB Server2 CRS files 2+2 RAID 10 1 D21 5 GB Server2 Datafiles, control files 4+1 RAID 5 10 D1 to D TB Server2 2.5 TB Server3 Temp files 4+1 RAID 5 2 Redo logs 2+2 RAID 10 2 D GB Server2 D GB Server3 D GB Server2 D GB Server3 Oracle dnfs Client configuration The Oracle 11g Database needs to be configured to use the Oracle 11g dnfs Client ODM disk libraries. This is a one-time operation and once set, the database will be configured to use the Oracle-optimized, native Oracle dnfs client, rather than the operating system s hosted NFS client. The standard ODM library was replaced with one that supports dnfs Client. Figure 5 shows the commands that enable the dnfs Client ODM library. Figure 5. Enable the dnfs Client ODM library We configured the Oracle dnfs Client for the virtual environments. We mounted the Oracle file systems and made them available over regular NFS mounts. Oracle dnfs Client used the oranfstab configuration file to determine the mount point settings for the NFS storage devices. Figure 6 shows an extract from the oranfstab file used for this solution. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 16

17 Figure 6. Extract from oranfstab configuration file Once configured, the management of dnfs mount points and load balancing is controlled from the oranfstab and not by the OS. 17

18 Configuring Oracle databases Database and workload profile Table 5 details the database and workload profile for this solution. Table 5. Database and workload profile Profile characteristic Database type Database size Oracle RAC Oracle SGA for each node Database performance metric Details OLTP 2 TB 4 nodes 12 GB TPM Database read/write ratio 60/40 User scaling 50 to 250, in increments of 50 (node scalability testing) 250 to 750, in increments of 100 (FAST Suite testing) Oracle database schema This solution applied a simulated OLTP workload by scaling users using Swingbench. We populated a 2 TB database. One 1 TB data database was accessed by different sessions that were running on the four nodes: ovm-01, ovm-02, ovm-03, and ovm-04. Another 1 TB schema data was left idle to simulate a more realistic skew in the dataset. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 18

19 Configuring FAST Cache on EMC VNX7500 Overview FAST Cache uses Enterprise Flash drives to add an extra layer of high-speed cache between DRAM cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the most active data is served from high-performing Flash drives and can reside on this faster medium for as long as is needed. FAST Cache is most effective when application workloads exhibit high data activity skew. This is where a small subset of data is responsible for most of the dataset activities. Fast Cache is more effective when the primarily block reads and writes are small, fits within the 64K FAST Cache track. The storage system is able to take advantage of such data skew by dynamically placing data according to its activity. For those applications whose datasets exhibit a high degree of skewing, FAST Cache can be assigned to concentrate a high percentage of application IOPs on Flash capacity. This section discusses using FAST Cache and outlines the main steps we carried out to configure and enable FAST Cache for this solution. You can perform the configuration steps using either the Unisphere GUI or the Unisphere command line interface (CLI). For further information about configuring FAST Cache, see Unisphere Help in the Unisphere GUI. Analyze the application workload Before you decide to implement FAST Cache, you must analyze the application workload characteristics. Array-level tools are available to EMC field and support personnel for determining both the suitability of FAST Cache for a particular environment and the right size for FAST Cache to configure. Contact your EMC sales teams for guidance. Whether a particular application can benefit from using FAST Cache, and what the optimal cache size should be, depends on the size of the application s active working set, the access pattern, the IOPS requirement, the RAID type, and the read/write ratio. As indicated in the Technology overview > EMC FAST Cache section of this white paper, the workload characteristics of OLTP databases make them especially suitable for using FAST Cache. For further information, see the white papers: EMC FAST Cache A Detailed Review and Deploying Oracle Database 11g Release 2 on EMC Unified Storage. For this solution, we performed an analysis using the EMC array-level tools, which recommended using FAST Cache and four 200 GB Flash drives as the optimal configuration. Create and enable/disable FAST Cache FAST Cache best practices for Oracle For details about creating, enabling, or disabling FAST Cache for a pool, see the white paper EMC FAST Cache A Detailed Review. The following are recommended practices: Disable FAST Cache on pool/luns that do not require it. Size FAST Cache appropriately, depending on the application s active dataset. 19

20 Disable FAST Cache on pool/luns where Oracle online redo logs reside. Never enable FAST Cache on archive logs, because these files are never overwritten and are rarely read back (unless you need to recover the database). EMC recommends that you enable FAST Cache for the Oracle datafiles only. Oracle archive files and redo log files have a predictable workload composed mainly of sequential writes. The array s write cache and assigned HDDs can efficiently handle these archive files and redo log files. Enabling FAST Cache on these files is neither beneficial nor cost effective. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 20

21 Configuring FAST VP on EMC VNX7500 Overview FAST VP is a game-changing technology that provides compelling advantages over traditional tiering options. It combines the advantages of automated storage tiering with Virtual Provisioning to optimize performance and cost while radically simplifying management and increasing storage efficiency. Like FAST Cache, FAST VP works best on datasets that exhibit a high degree of skew. FAST VP is very flexible and supports several tiered configurations, such as single tiered, multitiered, with or without a Flash tier, and FAST Cache support. Adding a Flash tier can locate hot data on Flash storage in 1 GB slices. FAST VP can be used to aggressively reduce TCO and/or to increase performance. A target workload that requires a large number of performance tier drives can be serviced with a mix of tiers, and a much lower drive count. In some cases, you can achieve an almost two-thirds reduction in drive count. In other cases, performance throughput can double by adding less than 10 percent of a pool s total capacity in Flash drives. You can use FAST VP in combination with other performance optimization software, such as FAST Cache. A common strategy is to use FAST VP to gain TCO benefits while using FAST Cache to boost overall system performance. There are other scenarios where it makes sense to use FAST VP for both purposes. This paper discusses considerations for an optimal deployment of these technologies. For further information on Fast VP algorithm and policies, see EMC FAST VP for Unified Storage Systems. Tiering policies FAST VP includes the following tiering policies: Start high then auto-tier (new default policy) Auto-tier Highest available tier Lowest available tier No data movement Start high then auto-tier (new default policy) Start high then auto-tier is the default setting for all pool LUNs on their creation. Initial data placement is on the highest available tier and then data movement is subsequently based on the activity level of the data. This tiering policy maximizes the initial performance and takes full advantage of the most expensive and fastest drives first, while providing subsequent TCO by allowing less active data to be tiered down, making room for more active data in the highest tier. When a pool has multiple tiers, the start high then auto-tier design is capable of relocating data to the highest available tier regardless of the drive type combination. Also, when adding a new tier to a pool, the tiering policy remains the same and there is no need to manually change it. 21

22 Auto-tier FAST VP relocates slices of these LUNs based on their activity level. Slices belonging to LUNs with the auto-tier policy have second priority for capacity in the highest tier in the pool after the LUNs are set to the highest tier. Highest available tier Select the highest available tier setting for those LUNs which, although not always the most active, require high levels of performance whenever they are accessed. FAST VP prioritizes slices of a LUN with the highest available tier selected above all other settings. Slices of LUNs set to the highest tier are rank ordered with each other according to activity. Therefore, in cases where the sum total of LUN capacity set to the highest tier is greater than the capacity of the pool s highest tier, the busiest slices occupy that capacity. Lowest available tier Select the lowest available tier for LUNs that are not performance-sensitive or response time-sensitive. FAST VP maintains slices of these LUNs on the lowest storage tier available, regardless of activity level. No data movement The no data movement policy may only be selected after a LUN has been created. FAST VP will not move slices from their current positions once the no data movement selection has been made. Statistics are still collected on these slices for use if and when the tiering policy is changed. Configure FAST VP In this solution, we set the Auto-Tiering policy to Scheduled. We configured the Data Relocation Schedule setting as Monday to Sunday, starting from 00:00 to 23:45, which determines the time window when FAST VP moves data between tiers. Note The Data Relocation Rate and Data Relocation Schedule are highly dependent on the real workload in a customer environment. Usually, setting the Data Relocation Rate to Low has less impact on the current running workload. Set the Tiering Policy for all LUNs containing datafiles to Auto-Tier, so that FAST VP can automatically move the most active data to Flash drive devices. For the details of FAST VP configuration, refer to EMC FAST VP for Unified Storage Systems A Detailed Review. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 22

23 Configuring Oracle virtual machines Overview Oracle VM Server for x86 is a Oracle server virtualization platform that makes enterprise applications easier to deploy and manage. As shown in Figure 7, the Oracle virtual machine environment consists of one or more Oracle VM servers that are grouped into a server pool and managed by Oracle VM Manager. Oracle VM Server allows hardware resources (CPUs, memory, networks, and storage) to become flexible resource pools from which virtual machines are built. Figure 7. Oracle VM environment overview Step 1: Prepare Oracle VM Server Oracle VM Server is a central component of Oracle VM. Oracle VM Server includes a virtual machine agent that enables the communication channel with the Oracle VM Manager for management purposes. After successful installation of the Oracle VM Server, the agent is running as shown in Figure 8. Refer to Oracle VM installation and upgrade Guide for detailed installation guidelines. 23

24 Figure 8. Oracle VM agent is running after installation Step 2: Install Oracle VM Manager Oracle VM Manager provides a graphical interface to create and manage the virtual infrastructure. After successful installation of the Oracle VM Manager, a login window to the web-based console is available as shown in Figure 9. Figure 9. Oracle VM Manager web console Refer to Oracle VM installation and upgrade Guide for detailed installation guidelines. Step3: Configure server pool The Oracle VM Manager connects with one or more Oracle VM servers to create, configure, and manage virtual resources. You can configure one or more Oracle VM servers in a server pool. Within a server pool, you can live-migrate virtual machines for load-balancing purposes or for scheduled maintenance. If a pool member disappears, you can recover its virtual machines and bring them back up on another Oracle VM server, because all necessary resources are available on the shared storage. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 24

25 In this solution, we configured two Oracle VM server pools consisting of three Oracle VM servers providing database and application services. Each server pool was assigned with virtual IPs, as shown in Figure 10. Figure 10. Oracle VM server pool configuration Step 4: Configure network The networking infrastructure in the Oracle VM environment is composed of a 1 Gb public network for accessing virtual machine, a 10 Gb IP storage network for virtual machines to access data movers, and a 10Gb network among virtual machines for RAC nodes interconnect. Creating an Oracle VM network involves mapping available network ports to a set of logical Ethernet networks. An Oracle VM network can perform one or more of the following network functions: Server management Live migration Virtual machine Storage In this solution, we used two network functions: The network function server management was used to manage the virtual machine s public network. The network function virtual machine was used to manage the network traffic between different virtual machines in a server pool and to manage the connections from the virtual machine to external data movers. As shown in Figure 11, a total of six Oracle VM networks were created on the Oracle VM Server (OVS). Two networks were 1 Gb public networks which were created automatically for server management, managing the physical Oracle VM servers in two server pools. The other four Oracle VM networks were configured with next-generation, 10 GbE connectivity. vinterconnect was the private network dedicated to the Oracle cluster interconnects. 25

26 vstorage1 and vstorage2 were private networks serving the NFS storage array to two Oracle virtual machines deployed on it individually. vstorageresil was created for storage redundancy purpose, in order to demonstrate the multipath function of Oracle dnfs. Port (5) of the two OVS was used for RAC interconnect. 10 Gb network was used for RAC interconnect and IP storage connection. Figure 11. Oracle VM network configuration Figure 12 and Figure 13 show the detailed network configurations of the two Oracle VM servers where the four-node RAC is running. Figure 12. Ethernet port configuration of OVS 1 Figure 13. Ethernet port configuration of OVS 2 Step 5: Configure Oracle virtual machines You can create Oracle VM guest operating systems in paravirtualized or hardware virtualized mode. In the paravirtualized mode, you can recompile the kernel of the guest operating system to make it aware of the virtual environment. This allows the paravirtualized guest to run at near native speed, because most memory, disk, and network accesses are optimized for maximum performance. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 26

27 Note RAC for Linux x86 and Linux x86_64 are certified on Oracle VM, but in the Paravirtualized (PV) mode only (Guest OS and drivers). In this solution, Xen PVM was chosen as shown in Figure 14. See Oracle Support Doc ID for details. Figure 14. Oracle VM type For this solution, we first created one paravirtualized virtual machine and configured with the prerequisites for the Oracle software (see Table 6), including: Virtualization mode Operating system RPM packages Kernel configuration OS users Supporting software We then cloned the virtual machines to create the four-node RAC environment. Table 6. Part CPU Memory Oracle virtual machine configuration Description 14 vcpus 32 GB Operating system Oracle Linux Server release 6.2 Kernel Virtualization mode Network interfaces OS user (user created and password set) el6uek.x86_64 paravirtualized Eth0: public/management IP network Eth1 (10 Gb): dedicated to cluster interconnect Eth2 (10 Gb): dedicated to NFS connection to Data Mover 2 Eth3 (10 Gb): dedicated to NFS connection to Data Mover 3 Eth4 (10 Gb): dedicated to NFS connection to Data Mover 2 as redundancy Eth5 (10 Gb): dedicated to NFS connection to Data Mover 3 as redundancy Username: oracle UserID:

28 Part OS groups RPM packages installed (as Oracle prerequisites) Disk configuration System configuration (as Oracle prerequisites) Description Group: oinstall GroupID:1000 Group: dba GroupID:1031 See the relevant Oracle installation guide. 30 GB virtual disk for root, /tmp, and the swap space 15 GB virtual disk for Oracle 11g R2 Grid and RAC Database binaries Note As of Oracle Grid Infrastructure , allow for an additional 1 GB of disk space per node for the Cluster Health Monitor (CHM) Repository. By default, this resides within the Grid Infrastructure home. See the following Oracle Installation Guides: Oracle Real Application Clusters Installation Guide 11g Release 2 (11.2) for Linux Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 28

29 Node scalability test Test objective Test procedure The goal of this test is to demonstrate the performance scalability, with both nodes and users scaled out on Oracle RACs with dnfs and 10 GbE in a virtualized environment. We ran an OLTP-like workload against a single node. We then added users and nodes to show the scalability of both node and user. We used Swingbench to generate the OLTP-like workload. The testing included the following steps: 1. Ran the workload on the first node by gradually increasing the number of concurrent users from 50 to 250 in increments of Added the second node into the workload, and ran the same workload as the previous step on each node separately, which means the total users scaled from 100 (50 on each node) to 500 (250 on each node). 3. Repeated the previous two steps after adding the third and fourth nodes separately. 4. For each user iteration, we recorded the front-end IOPS from Unisphere for VNX, the TPM from Swingbench, and the performance statistics from Oracle AWR reports. Notes Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance varies based on many factors. Therefore, you cannot use this workload as a substitute for a specific environment s application benchmark when making critical capacity planning or product evaluation decisions. The testing team obtained all performance data in a rigorously controlled environment. Results of other operating environments can vary significantly. EMC Corporation does not guarantee that a user can achieve similar performance demonstrated in TPM. 29

30 Test result The Cache Fusion architecture of Oracle RAC immediately uses the CPU and memory resources of the new node(s). Thus, we can easily scale out the system CPU and memory resource without affecting the online users. It provides a scalable computing environment that supports the application workload. Figure 15 shows the TPM Swingbench recorded during the node scalability testing, scaling from both nodes and concurrent users. We scaled the RAC database nodes from one to four. In each RAC configuration, we ran the Swingbench workload with 50, 100, 150, 200, and 250 users on each node. We observed a near-linear scaling TPM from Swingbench as the concurrent user load increased along with the scale of nodes. The chart illustrates the benefits of using EMC VNX7500 storage with Oracle RAC and dnfs for achieving a scalable OLTP environment. Oracle RAC provides not only horizontal scaling, but also guaranteed continuous availability. Figure 15. Node scalability test EMC FAST Suite can automatically optimize the storage to ensure the highest system performance at all times, which helps to improve the system efficiency. FAST Cache working with FAST VP not only boosts the application performance but also provides improved TCO of the whole system. See the FAST Suite test section for test results of enabling FAST Suite on the OLTP workload. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 30

31 FAST Suite test FAST Suite and manual tiering comparison Manual tiering involves a repeated process that can take up to nine hours or more to complete each time. In contrast, both FAST VP and FAST Cache operate automatically, eliminating the need to manually identify and move or cache the hot data. As shown in Figure 16, configuring FAST Cache is a one-off process taking 50 minutes or less; hot and cold data is then cached in and out of FAST Cache continuously and automatically. Figure 16. FAST Suite and manual tiering comparison Note The time stated for configuring FAST VP is a conservative estimate. For details about configuring FAST VP, see the Configuring FAST VP on EMC VNX section of this white paper. FAST Cache test FAST Cache boosts the overall performance of the I/O subsystem. Fast Cache enables applications to deliver consistent performance by absorbing heavy read/write loads at Flash drive speeds. We configured four 200 GB Flash drives for FAST Cache. This provided a FAST Cache of 400 GB. We enabled FAST Cache for the storage pool that contains the database datafiles. FAST Cache warm-up FAST Cache requires some warm-up time before the I/O subsystem can achieve a high performance. Figure 17 tracks the FAST Cache read/write hit ratio of the storage pool that stores the datafiles. 31

32 Figure 17. FAST Cache warm-up period FAST Cache was empty when it was initially created. During the warm-up period, as more hot data was cached, the FAST Cache hit rate increased gradually. In this test, the hit ratio increased to 91 percent after a warm-up period of approximately six hours. When the locality of the active data changes, it is required to warm up the new data. This process is a normal, expected behavior and is fully automatic. FAST Cache test procedure To test the performance enhancement provided by FAST Cache, we ran the Swingbench workload on all of the four RAC nodes concurrently, with and without FAST Cache enabled. The test procedure includes the following steps: 1. Baseline testing: a. Ran the workload against the database from four RAC nodes at the same time without FAST Cache, and scaled it from 250 concurrent users to 750 on each node. The active data size was 1 TB, which was deployed on SAS drives only. b. Monitored the performance statistics, including average front-end IOPS and database TPM for each user iteration from Oracle AWR reports and Unisphere for VNX. 2. FAST Cache testing: a. Enabled FAST Cache on the storage array after the baseline testing, then ran the same workload and collected performance statistics as the baseline. Along with the number of users, the running workload increased. FAST Cache cached more hot data, which improved the performance. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 32

33 b. After all the FAST Cache testing finished, compared the performance data with the baseline to determine how much the performance enhancement FAST Cache can offer. FAST VP test We created a two-tier FAST VP with a mixed storage pool, consisting of five Flash drives and 40 SAS drives on VNX7500. FAST VP automatically relocated the LUN data from one disk tier to another within a pool. FAST VP moving data across tiers Initially, all datafiles were placed on SAS devices. With the workload running against the database from four RAC nodes at the same time for a few hours, FAST VP monitored and moved data, as shown in Figure 18. Figure 18. FAST VP in progress of moving data As long as the storage pool followed the applied FAST VP policy, the load continued to ensure the sustainability of the performance levels observed, as shown in Figure 19. Figure 19. FAST VP in a steady state FAST VP test procedure In order to test the performance enhancement of FAST VP, we ran the same workload as that used in FAST Cache testing against the database with FAST VP enabled. The test procedure included the following steps: 1. Ran the same workload and collected performance statistics as the baseline. 33

34 2. After all the FAST VP testing finished, compared the performance data with the baseline to determine how much the performance enhancement FAST VP can offer. The results of these tests are detailed in the Test result section. Note The synthetic benchmark used during testing is more uniformly random than real-world applications, which tends to have greater locality of reference. Customer environments have more inactive data. As a result, we believe that most organizations are able to use less Flash drives capacity to achieve similar, if not better, performance benefits with FAST VP. Customers can achieve additional cost savings if introducing the NL-SAS drives to create the third layer as inactive data is down tiered to NL-SAS. In this solution, the 1 TB of inactive data is automatically moved to the third layer if the layer has been configured. FAST Suite test FAST Suite is the combination of FAST Cache and FAST VP. To demonstrate the advantages of FAST Cache in absorbing random I/O bursts and the benefits of FAST VP s auto-tiering feature in performance improvement and cost saving, we designed the following two test scenarios: Five Flash drives in FAST VP and four Flash drives for FAST Cache. Five Flash drives in FAST VP and two Flash drives for FAST Cache. Note Refer to the Analyze the application workload section to appropriately size the Flash drives for FAST Cache. To understand why we used five Flash drives for FAST VP, refer to EMC VNX Unified Best Practice For Performance - Applied Best Practices Guide. The rule of thumb for tier construction on extreme performance Flash tier is 4+1 RAID 5. It yields the best performance versus capacity balance. In order to test the performance of FAST Suite, we ran the same workload as that used in the FAST Cache test scenario and FAST VP test scenario. FAST Suite test procedure Initially, we placed all datafiles on SAS devices and tested the FAST Suite with four Flash drives for Fast Cache and five Flash drives for FAST VP. 1. Enabled FAST Cache using four Flash drives and enabled FAST VP using five Flash drives. 2. Generated the workload against the database to warm up the FAST Cache so that it can reach a stable read/write hit ratio and FAST VP s moving data was also stable. 3. Generated the workload against the database to ensure that FAST VP was monitoring and moving data. 4. Increased the number of users running transactions at intervals to determine how the database performed. 5. Monitored the performance of the database, and recorded the average frontend IOPS and database TPM for each user iteration. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 34

35 Then we tested the FAST Suite with two Flash drives for Fast Cache and five Flash drives for FAST VP: 6. Destroyed FAST Cache and enabled FAST Cache again using two Flash drives. 7. Restored the database. 8. Repeated step 2 to step 5. Test result FAST Suite effects on database transactions per minute This section compares the database TPM for each test scenario mentioned previously, which includes the following: Baseline testing FAST Cache-only testing using four 200 GB Flash drives, configured with RAID 1. FAST VP-only testing using five 200 GB Flash drives, configured with RAID 5. FAST Suite combination testing using four 200 GB Flash drives for FAST Cache and five 200 GB Flash drives for FAST VP. FAST Suite combination testing using two 200 GB Flash drives for FAST Cache and five 200 GB Flash drives for FAST VP. Figure 20 shows the TPM recorded during the period that the Swingbench workload scaled from 250 to 750 users on each node. This chart shows that the number of transactions processed was much higher when introducing EMC FAST Suite. Figure 20. TPM with and without FAST Suite 35

36 When enabling FAST VP, five Flash drives were added to the data pool as RAID 5, the TPM increased by about 20 percent and stabilized at around 270,000, and the read response time was reduced by 50 percent. When enabling FAST Cache, we used four Flash drives. The TPM surged to around 500,000 and stabilized at that level. The response time was dramatically decreased to less than 10 ms. See the Wait statistics from Oracle AWR reports section for detailed analysis from the database side. The other two test results in Figure 20 show the performance of combining FAST Cache and FAST VP, which are two complementary technologies. When configuring four Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly higher than that of using four Flash drives for FAST Cache only, and the read response time was reduced by 36 percent accordingly. When configuring two Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly lower than when using four Flash drives for FAST Cache only, and the read response time was tripled. Notes When using FAST VP, customers can achieve additional cost savings if NL-SAS drives are added to create the third layer. The higher tiers fill to 90 percent of their available capacity, and cooler data is automatically migrated down to the lower tier as a result of hotter data displacing it. In this solution, we had 1 TB inactive data, which would automatically be moved to the third layer if the layer was configured, thus making space available on the SAS tier for other applications. Figure 21 shows a different view of TPM comparison. The performance improvement of using FAST Suite is clear. Figure 21. TPM comparison EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 36

37 FAST Suite performance on read response time Figure 22 depicts the significant improvement of read response time when enabling EMC Fast Suite compared with the baseline. When using FAST VP, the response time decreased from ms to 61.1 ms. In addition, if you have a number of NL-SAS drives, you can use them as a capacity layer, which reduces your TCO by moving cold data to the capacity layer. When enabling FAST Cache, the response time decreased from ms to 6.28 ms. When using both FAST Cache and FAST VP, we reduced the read response time from ms to ms using seven Flash drives, and reduced the read response time from ms to 4.28 ms using nine Flash drives. Figure 22. Read response time comparison Wait statistics from Oracle AWR reports Oracle foreground wait statistics highlight potential bottlenecks in Oracle RAC environments. Figure 23 and Figure 24 show the excerpt of top wait events from the RAC Automatic Workload Repository (AWR) reports and compare the waits for the baseline and FAST-only test as well as the FAST Suite combination test respectively. The figures show that the I/O performance was greatly improved when using FAST Cache or FAST Suite combination - the average wait time for db file sequential read decreased dramatically. Because of the increase of supported concurrent user transactions, the commit operations grew rapidly. However, the average wait time for log file sync decreased by almost 50 percent. 37

38 Figure 23. AWR reports comparison between baseline and FAST Cache-only tests Figure 24. AWR reports comparison between baseline and FAST Suite combination tests EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 38

39 Statistics from Unisphere for VNX Figure 25 shows the increase in average IOPS for the datafile systems. The IOPS increased over 200 percent when enabling FAST Suite. Figure 25. IOPS comparison The I/O statistics generated from Unisphere, the TPM from Swingbench (Figure 21), and the read response time (Figure 22) demonstrated the advantages of enabling FAST Suite from different perspectives. 39

40 dnfs clonedb test Test objectives Test procedures Customers often need to clone a production database to develop and test new application patches. A new feature called dnfs clonedb enables users to clone test databases instantaneously. To quickly provision a test database based on the snapshot of database file systems created by EMC VNX SnapSure using the dnfs clonedb feature, the testing team performed the following steps: 1. Installed Oracle database software in the test environment. 2. Ran the command as shown in Figure 26 to enable dnfs in the test/development environment and create a dnfs configuration file, as shown in the Oracle dnfs Client configuration section. Figure 26. Enable dnfs 3. To take a hot backup, a. Put the database in hot backup mode with the following command in SQL*PLUS: alter database begin backup; b. Created the VNX SnapSure checkpoint against the database file systems with the following commands. fs_ckpt data1 -name ck_data1 -Create pool=save_pool fs_ckpt data2 -name ck_data2 -Create pool=save_pool Note If using SnapSure to create user checkpoints of the primary file system, place SavVol on separate disks when possible and avoid enabling FAST Cache on SavVol. For details, see Applied Best Practices Guide: EMC VNX Unified Best Practices for Performance. c. Take the database out of hot backup mode with the following command in SQL*PLUS. alter database end backup; 4. Mounted the VNX SnapSure checkpoint to the target virtual database server. 5. Generated the backup control file script from the production database with the following command in SQL*PLUS. alter database backup controlfile to trace; 6. Copied the spfile and the backup control file from the production database to the test environment and made the necessary changes. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 40

41 Note To avoid failure of the dbms_dnfs.clonedb_renamefile, you must set clonedb=true in the initialization parameter file for the cloned database. 7. Started up the cloned database instance with the nomount option and ran the modified backup control file script. 8. Ran the dbms_dnfs.clonedb_renamefile procedure for each datafile in the cloned database. For example: declare begin dbms_dnfs.clonedb_renamefile('/u02/oradata/racdb784/ soe3_13.dbf.dbf', '/clonedb/uc784/dnfs784/soe3_13.dbf.dbf.dbf'); end; 9. Recovered database with the following command in SQL*PLUS. recover database using backup controlfile until cancel; This command asks the archivelogs for the period when the backup was taken and input those file names. 10. Opened the cloned database with the resetlogs option. Test results When the cloned database is up and running, we can perform read and write activities on the test database. When the workload running is started, storage consumption of the cloned database grows with the speed at which the data is modified. To verify the function of the dnfs clonedb database, we generated the workload using Swingbench against the cloned database as shown in Figure 27. Figure 27. Workload against the dnfs clonedb database 41

42 Resilience test Test objectives The goal is to outline the availability and resilience of the dnfs architecture by demonstrating the database availability during the physical NIC failure and a data mover panic. Up to four network paths defined in the oranfstab file for an NFS server can be used with Oracle dnfs features. The dnfs Client performs load balancing across all specified paths. If one of the paths fails, dnfs reissues I/O commands over any other remaining paths. Test procedures Physical NIC failure We manually shut down the NIC to simulate physical NIC failure. The test procedure included: 1. Configured the database with two paths to two data movers separately. 2. Ran the Swingbench workload against the first node with 100 users. 3. Shut down the physical NIC on the virtual machine server to disconnect the resiliency path route to the two data movers as shown in Figure 28. Figure 28. Physical NIC shutdown 4. Monitored the alert log for warnings as shown in Figure 29. Figure 29. dnfs path down messages 5. After a few seconds, started up the physical NIC as shown in Figure 30. Figure 30. Physical NIC startup 6. Monitored the alert log for warnings as shown in Figure 31. Figure 31. dnfs path up messages 7. Waited for the Swingbench workload to complete. EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM, 42

43 Data mover panic We manually failed over one data mover to the standby one to simulate a data mover panic. 1. Deployed the database datafiles on two file systems across two data movers: server_2 and server_3. 2. Ran the Swingbench workload against the first node with 100 users. 3. Failed the data mover server_2 over to server_5, as shown in Figure 32. Figure 32. Data mover failover This command activates the server_5 (standby data mover) to take over server_2 (the primary data mover). 4. Verified that the standby data mover server_5 replaced the primary data mover server_2 as shown in Figure 33. Figure 33. Data mover status check after failover 5. Failed back the data mover server_2 as shown in Figure 34. Figure 34. Data mover failback 6. Verified that the data mover server_2 is being restored to the primary data mover successfully as shown in Figure