EMC VSPEX ORACLE COMPUTING Oracle Database Virtualization with VMware vsphere and EMC XtremIO

DESIGN GUIDE EMC VSPEX ORACLE COMPUTING Oracle Database Virtualization with VMware vsphere and EMC XtremIO Enabled by EMC Data Protection EMC VSPEX Abstract This Design Guide describes how to best design and size virtualized Oracle Database 11g or 12c resources in an EMC VSPEX Proven Infrastructure using VMware vsphere on EMC XtremIO with EMC Data Protection. July 2015

Copyright 2015 EMC Corporation. All rights reserved. Published in the USA. Published July 2015 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vsphere and EMC XtremIO Design Guide Part Number H14312 2 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Contents Contents Chapter 1 Introduction 7 Purpose of this guide... 8 Business value... 8 Audience... 9 Terminology... 10 Chapter 2 Before You Start 11 Deployment workflow... 12 Essential reading... 13 Chapter 3 Solution Overview 14 Overview... 15 EMC VSPEX Proven Infrastructure... 15 Solution architecture... 16 Overview of key technologies... 19 Oracle Database 11g R2 and 12c R1... 19 VMware vsphere 5.5... 19 EMC XtremIO 4.0... 20 XtremIO Management Server... 22 Red Hat Enterprise Linux 6.5... 22 EMC backup and recovery solutions... 22 Chapter 4 Choosing a VSPEX Proven Infrastructure 24 Overview... 25 Step 1: Evaluate the customer use case... 25 Step 2: Design the application architecture... 26 Step 3: Select the right VSPEX Proven Infrastructure... 26 Chapter 5 Solution Design Considerations and Best Practices 28 Overview... 29 Designing the network... 29 Overview... 29 Network best practices... 29 VMware vsphere network best practices... 29 Recommended network design... 30 Designing the server... 31 EMC VSPEX Oracle Computing: Oracle Database Virtualization 3

Contents Overview... 31 Server best practices... 31 Validated server design... 32 Designing the storage layout... 32 Overview... 32 XtremIO X-Brick layout... 32 XtremIO X-Brick scalability... 33 Validated XtremIO server virtualization... 34 vsphere storage virtualization best practices... 35 Designing an Oracle database... 35 Overview... 35 Oracle storage layout... 36 Oracle design considerations... 36 Oracle licensing considerations... 37 Implementing EMC Data Protection... 37 Chapter 6 Solution Testing and Validation 38 Overview... 39 Test methodology and reference workload... 39 Test methodology... 39 Reference workload... 39 OLTP workload test performance results... 41 OLTP sizing test results... 41 XtremIO snapshot test performance results... 44 Verification methodologies... 46 Understand key metrics... 46 Sizing guidelines... 47 Overview... 47 Using the Customer Sizing worksheet... 47 Determining the server resource requirements... 48 Determining the storage resource requirements... 49 Chapter 7 Reference Documentation 51 EMC documentation... 52 Other documentation... 52 Oracle documentation... 52 Appendix A Qualification Worksheet 53 VSPEX qualification worksheet for virtualized Oracle databases... 54 Gather the information from the customer s Oracle databases example... 54 4 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Figures Contents Figure 1. VSPEX Proven Infrastructure... 16 Figure 2. Architecture of the validated infrastructure... 17 Figure 3. XtremIO snapshot... 21 Figure 4. XtremIO FC network example... 30 Figure 5. XtremIO scalability... 33 Figure 6. EMC XtremIO volume configuration and mapping... 34 Figure 7. Figure 8. Figure 9. Figure 10. Workload comparison for a single-instance Oracle database with different virtual machine configurations... 42 Single-instance Oracle database CPU utilization with mixed workloads43 Single-instance Oracle Database performance with and without a snapshot during OLTP workload... 44 Response time for a single-instance Oracle Database with and without a snapshot during OLTP workload... 45 Figure 11. Single-instance Oracle database performance... 45 Figure 12. init.ora Parameters from the AWR Report... 55 Figure 13. IOStat by function summary from the AWR Report... 56 Figure 14. Foreground Wait Event from the AWR report... 56 Tables Table 1. Terminology... 10 Table 2. VSPEX workflow for virtualized Oracle database deployment... 12 Table 3. Hardware resources... 18 Table 4. Software resources... 18 Table 5. VSPEX Proven Infrastructure selection steps... 25 Table 6. VSPEX qualification worksheet for virtualized Oracle databases guidelines... 25 Table 7. Select the right VSPEX Proven Infrastructure... 27 Table 8. Server hardware... 32 Table 9. Oracle storage design on XtremIO X-Brick... 36 Table 10. Reference virtual Oracle server characteristics (OLTP)... 40 Table 11. Reference virtual Oracle server characteristics (DSS)... 40 Table 12. Example virtual server resource requirements... 41 Table 13. Performance observation during an OLTP workload execution with 32 vcpus... 43 Table 14. High-level steps for application verification... 46 Table 15. Qualification worksheet example... 47 Table 16. Compute resources validated with a typical OLTP workload... 48 Table 17. Required virtual machine resources example (for OLTP)... 49 EMC VSPEX Oracle Computing: Oracle Database Virtualization 5

Contents Table 18. Compute resources validated with a typical DSS workload... 49 Table 19. Required virtual machine resources example (for DSS)... 49 Table 20. VSPEX qualification worksheet for virtualized Oracle databases... 54 6 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 1: Introduction Chapter 1 Introduction This chapter presents the following topics: Purpose of this guide... 8 Business value... 8 Audience... 9 Terminology... 10 EMC VSPEX Oracle Computing: Oracle Database Virtualization 7

Chapter 1: Introduction Purpose of this guide Business value EMC VSPEX Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, greater simplicity, more choice, higher efficiency, and lower risk. VSPEX provides partners with the ability to design and implement the virtual assets that are required to support a fully integrated virtualization solution for an Oracle relational database management system (RDBMS) on a VSPEX private cloud infrastructure. The VSPEX for virtualized Oracle infrastructure provides customers with a modern system capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This Design Guide describes how to plan and design a VSPEX Proven Infrastructure for VMware vsphere virtualized Oracle Database 11g or 12c. It provides deployment examples for virtual Oracle Database 11g or 12c on EMC XtremIO storage arrays. The guide assumes that a VSPEX Private Cloud already exists in the customer environment. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment. Note: The Oracle versions for this solution are Oracle Database 11g Release 2 (11.2.0.4) and Oracle Database 12c Release 1 (12.1.0.2). We refer to these releases as Oracle 11g R2 and Oracle 12c R1 throughout the document. This guide also describes how to use the VSPEX Sizing Tool to size Oracle Database 11g or 12c on a VSPEX Proven Infrastructure, how to use best practices to efficiently allocate resources, and how to use all the benefits that VSPEX offers. The EMC Data Protection solutions for Oracle server data protection are described in a separate document, EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11g R2 Design and Implementation Guide. Database management systems software is used by many different types of businesses. Despite the increasing market share of other data management tools, growth in sales is expected to continue and accelerate as customers continue to diversify their infrastructures and supporting technologies and use more appliances and configurations. This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the XtremIO series, combined with Oracle, can bring to customers who often have growing, isolated IT environments running server-centric applications. 8 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 1: Introduction This VSPEX solution is designed to meet Oracle database challenges and enable customers to increase performance, scalability, reliability, and automation. By consolidating database applications on XtremIO, customers can consolidate a single centralized storage platform to more effectively manage the exploding data growth that is challenging businesses today. This solution has been sized and proven by EMC to: Deploy systems faster, saving time and effort with EMC Proven Solutions Increase performance and scalability out of the box Minimize storage requirements and reduce costs Audience This guide is intended for internal EMC personnel and qualified VSPEX partners. It assumes that VSPEX partners who intend to deploy this solution are: Qualified by EMC to sell, install, and configure the XtremIO family of storage systems Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures Certified to sell VSPEX Proven Infrastructure Partners who plan to deploy this solution must also have the necessary technical training and background to install and configure: VMware vsphere 5.5 Red Hat Enterprise Linux (RHEL) 6.5 Oracle Database 11g R2 or 12c R1 This guide provides external references where applicable. EMC recommends that partners who implement this solution are familiar with these documents. For details, refer to Essential reading and Reference Documentation. EMC VSPEX Oracle Computing: Oracle Database Virtualization 9

Chapter 1: Introduction Terminology Table 1 lists the key terms used in this guide. Term Table 1. Automatic Workload Repository (AWR) Decision Support System (DSS) Online transaction processing (OLTP) Process Global Area (PGA) Reference workload Raw device mapping (RDM) System Global Area (SGA) Storage Controller (SC) Virtual Machine Disk (VMDK) Virtual Machine File System (VMFS) XtremIO Management Server (XMS) XtremIO X-Brick Terminology Definition AWR is a powerful monitoring utility bundled with Oracle Database 10g and later releases. DSSs are a specific class of computerized information systems that supports business and organizational decision-making activities. OLTP is a class of information systems that facilitate and manage transaction-oriented applications, typically for data entry and retrieval transaction processing. PGA is memory dedicated to an operating process or thread that is not shared by other processes or threads on the system. The reference workload is defined as the reference virtual machine with the workload characteristics indicated in this Design Guide. By comparing the customer s actual usage to this reference workload, you can determine which reference architecture to choose as the basis for the customer s VSPEX deployment. RDM allows the virtual infrastructure to connect a physical device directly to a virtual machine. SGA is a group of shared memory structures containing data and control information from one Oracle database instance. Storage Controller is the computer component of the XtremIO storage array. SCs are used for all aspects of data moving into, out of, and between XtremIO arrays. VMware VMDK is an open file format that is a container for virtual hard disk drives for virtual machines. VMware VMFS is a cluster file system that uses storage virtualization for multiple installations of VMware ESX server. XMS is used to manage the XtremIO array and deployed as a virtual machine using an Open Virtualization Alliance (OVA) package. An X-Brick is a specialized configuration of the XtremIO All-Flash Array that includes 25 SSD drives for this solution. 10 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 2: Before You Start Chapter 2 Before You Start This chapter presents the following topics: Deployment workflow... 12 Essential reading... 13 EMC VSPEX Oracle Computing: Oracle Database Virtualization 11

Chapter 2: Before You Start Deployment workflow Refer to the process in Table 2 to design and implement a VSPEX workflow for your virtualized Oracle database solution. Table 2. VSPEX workflow for virtualized Oracle database deployment Step Action 1 Use the VSPEX qualification worksheet for virtualized Oracle databases to collect user requirements. You will find a one-page sizing worksheet in Appendix A of this guide. 2 Use the EMC VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for your Oracle Database 11g or 12c solution, based on the user requirements collected in Step 1. For more information, refer to the VSPEX Sizing Tool on the EMC Business Value Portal. Note: You need to register the first time you access the tool. 3 Use this Design Guide to determine the final design for your VSPEX solution. Note: Consider the requirements for all applications, not only the Oracle database. 4 Choose and order the VSPEX Proven Infrastructure components, including the XtremIO array, server systems, and network switches. Refer to the appropriate VSPEX Proven Infrastructure document in Essential reading for guidance. 5 Deploy and test your VSPEX solution. Refer to the appropriate VSPEX Implementation Guide in Essential reading for guidance. 12 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 2: Before You Start Essential reading Before implementing the solution described in this document, EMC recommends that you read the following documents, which are available from the VSPEX section in the EMC Community Network or from EMC.com and the VSPEX Partner Portal. If you do not have access to a document, contact your EMC representative. EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vsphere and EMC XtremIO Implementation Guide Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5 EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11g R2 Design and Implementation Guide EMC VSPEX Oracle Computing: Oracle Database Virtualization 13

Chapter 3: Solution Overview Chapter 3 Solution Overview This chapter presents the following topics: Overview... 15 EMC VSPEX Proven Infrastructure... 15 Solution architecture... 16 Overview of key technologies... 19 14 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 3: Solution Overview Overview This chapter provides an overview of the VSPEX Oracle Database 11g or 12c solution on XtremIO. The solution has been designed and proven by EMC to provide the server virtualization, network, storage, and backup resources to support reference architectures for the specialized configurations of the XtremIO All-Flash Array described in this section. This solution has the following benefits: EMC VSPEX Proven Infrastructure Enables consolidation of types of two database workloads, OLTP and DSS Provides storage designed for both low-latency transaction I/O and highthroughput analytic workloads Provides new levels of speed and provisioning agility to virtualized environments with space-efficient snapshots, inline copy deduplication, thin provisioning, and accelerated provisioning via VMware vstorage APIs for Array Integration (VAAI) VSPEX Proven Infrastructures, as shown in Figure 1, are modular, virtualized infrastructures validated by EMC and delivered by EMC VSPEX partners. They include virtualization, server, network, storage, and data protection layers. Partners can choose the virtualization, server, and network technologies that best fit a customer s environment, while the highly available XtremIO storage systems and EMC data EMC VSPEX Oracle Computing: Oracle Database Virtualization 15

Chapter 3: Solution Overview protection technologies provide the storage and data protection layers. Figure 1. VSPEX Proven Infrastructure Solution architecture Refer to the VSPEX Proven Infrastructure Guide in Essential reading for information on configuring the required infrastructure components. Figure 2 shows the architecture validated for an Oracle Database 11g or 12c server overlay on a VSPEX infrastructure. The XtremIO storage array is an all-flash system based on scale-out architecture. The system uses building blocks, called X-Bricks, which can be clustered to grow performance and capacity as required. The solution supports block storage for virtual Oracle databases using one X-Brick. 16 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 3: Solution Overview Figure 2. Architecture of the validated infrastructure To validate this solution, we 1 completed the following tasks: Deployed the Oracle Database 11g or 12c servers as virtual machines on VMware vsphere 5.5 Deployed the XtremIO array in multiple configurations to support different resources for both OLTP and DSS workloads on Oracle databases Determined the recommended storage layout for the Oracle database and the virtual infrastructure pool in the XtremIO storage arrays The solution architecture includes the following: Storage layer One X-Brick with a total of 7.58 TB of usable physical capacity Oracle database layer Oracle Database 11g or 12c server with multiple differently-sized databases and snapshots Network layer SAN/IP switches that are designed to support virtualized environments 1 In this guide, we refers to the EMC Solutions engineering team that validated the solution. EMC VSPEX Oracle Computing: Oracle Database Virtualization 17

Chapter 3: Solution Overview Physical servers and virtualization layer Multiple ESXi servers enable a highperforming and virtualized approach to deploy Oracle databases Table 3 lists the hardware resources used in this solution. Table 3. Hardware resources Equipment Quantity Configuration Server 2 2x servers for the Oracle 11g or 12c environment with: 40 CPU cores at 2.393 GHz 192 GB RAM 2x Dual-port 1 Gb Ethernet NICs 2x Dual-port 8 Gb HBAs SAN switch 2 FC director-class switches LAN switch 2 10 GbE Storage 1 A single X-Brick XtremIO AFA with 25x 400 GB SSD drives (total physical capacity 7.58 TB) Table 4 lists the software resources used in this solution. Table 4. Software resources Software Version Configuration/Source Oracle Database 11.2.0.4/12.1.0.2 Oracle Database software Oracle Grid Infrastructure 11.2.0.4/12.1.0.2 Oracle Clusterware software Red Hat Enterprises Linux 6.5 OS for database servers VMware vsphere 5.5 VMware hypervisor VMware vcenter 5.5 vsphere management XtremIO 4.0 All-flash storage Swingbench 2.5 benchmark tool 18 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 3: Solution Overview Overview of key technologies This section provides an overview of the key technologies used in this solution: Oracle Database 11g R2 or 12c R1 VMware vsphere 5.5 EMC XtremIO 4.0 Red Hat Enterprise Linux 6.5 EMC data protection Oracle Database 11g R2 and 12c R1 Oracle Database 12c R1 is the latest version of Oracle database technology. Oracle 11g R2 and 12c R1 are available in a variety of editions that are tailored to meet your business and IT needs. In this guide, we consider: Oracle Database 11g Release 2 /12c Release 1 Standard Edition (SE) Oracle Database 11g Release 2 /12c Release 1 Enterprise Edition (EE) Oracle Standard Edition (Oracle SE) is an affordable, full-featured data management solution that is ideal for all companies. It is available on single or clustered servers and can be licensed on a maximum capacity of four processor sockets, regardless of core count. The SE license includes Oracle Real Application Clusters (RAC) as a standard feature with no additional cost. Oracle Enterprise Edition (Oracle EE) delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. Oracle Database EE supports advanced features, either included or as extra-cost options, that are not available with Oracle Database SE. For example, security features such as Virtual Private Database are included with Oracle Database EE, as well as data warehousing options such as partitioning and advanced analytics. Note: The Oracle database edition affects the licensing cost and the size and number of VMware ESXi clusters that you can configure. How to Find the Oracle Processor Core Factor Multipliers (Doc ID 1330016.1) on My Oracle Support provides more information about Oracle processor licensing. For Oracle Database 12c Enterprise Edition, Oracle Multitenant is a new feature that helps to reduce IT costs by consolidation, provisioning, upgrades, and more. Oracle Multitenant is supported by a new architecture that enables a single super database to hold many sub-databases. Oracle Multitenant is fully interoperable with Oracle RAC. VMware vsphere 5.5 VMware vsphere 5.5 transforms a computer s physical resources by virtualizing the CPU, RAM, hard disk, and network controller. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications in the same way as physical computers. EMC VSPEX Oracle Computing: Oracle Database Virtualization 19

Chapter 3: Solution Overview VMware High Availability (HA) provides easy-to-use and cost-effective high availability for applications running on virtual machines. The VMware vsphere vmotion and VMware vsphere Storage vmotion features of vsphere 5.5 enable the seamless migration of virtual machines and stored files from one vsphere server to another, with minimal or no performance impact. Coupled with VMware vsphere Distributed Resource Scheduler (DRS) and VMware vsphere Storage DRS, virtual machines have access to the appropriate resources at any point in time through load balancing of compute and storage resources. VMware Native Multipathing Plug-In (NMP) is the default module in vsphere that is used for multipathing. NMP provides a default path selection algorithm based on the array type and it associates the physical paths with a specific storage device or logical unit number (LUN). The specific details for handling path failover for a given storage array are delegated to a Storage Array Type Plug-In (SATP). The specific details for determining which physical path is used to issue an I/O request to a storage device are handled by a Path Selection Plug-In (PSP). SATPs and PSPs are sub-plug-ins within the NMP module. EMC XtremIO 4.0 The XtremIO storage array is an all-flash system with scale-out architecture. The system uses building blocks, called X-Bricks, that you can cluster together to grow performance and capacity as required. XtremIO uses flash storage to deliver value across the following main dimensions: Performance Latency and throughput remain consistent, predictable, and constant, regardless of how busy the system is. Latency within the array for an I/O request is typically far less than one millisecond (ms). Scalability Based on a scale-out architecture, single X-Brick is a building block of XtremIO. Multiple X-Bricks can be clustered together to provide additional performance or capacity. Performance scales linearly to ensure that two X-Bricks supply twice the IOPS, and four X-Bricks supply four times the IOPS of the single X-Brick configuration. The latency remains consistently low as the system scales out. Inline data reduction The core XtremIO engine implements content-based inline data reduction. XtremIO automatically reduces (deduplicates and compresses) data as the system processes it. This reduces the amount of data written to flash, improving the longevity of the media, and reducing cost. Volumes are always thin provisioned without any loss of performance, overprovisioning of capacity, or fragmentation. Data protection XtremIO uses a proprietary flash-optimized data protection algorithm, XtremIO Data Protection (XDP), which provides superior data protection while enabling performance that surpasses any existing RAID algorithms. Optimizations in XDP also result in fewer writes to flash media for data protection purposes. Functionality XtremIO supports high performance and space-efficient snapshots, inline data reduction, thin provisioning, and full vsphere VAAI, integration with support for Fibre Channel (FC) and iscsi protocols. 20 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 3: Solution Overview Simplicity Provisioning storage with XtremIO is as simple as deciding how large a LUN you want to create. You no longer need to select the RAID type, create a RAID group, or decide whether or not to enable thin provisioning or deduplication. These functions are already built into XtremIO. XtremIO also elevates writable snapshots beyond simple data protection. XtremIO snapshots are equivalent to production volumes for performance, property, and functions, which means that a snapshot in XtremIO can be considered the same as the production volume. Figure 3 shows how XtremIO works in an environment that demands large amounts of development, testing, or quality assurance (QA) data from a writable snapshot. Figure 3. XtremIO snapshot XtremIO snapshots provide users with the following benefits when using production data for development and test needs: Inherently writable, not read-only Built into metadata that is only needed for globally unique writes; entire metadata copies are not required as in other snapshot implementations Used as live production volumes without the need to create a writable or instantiate snapshot for read/write access Space-efficient and metadata-efficient EMC VSPEX Oracle Computing: Oracle Database Virtualization 21

Chapter 3: Solution Overview Each snapshot does not need a full metadata structure Common metadata is shared between production and snapshot Space is only used for new unique data blocks and associated metadata Deduplication and thin provisioning is always on Enables affordable consolidation Maximum performance, scalability, and economy Instant creation of a complete snapshot No impact on system performance No overhead from brute-force copies No metadata bloat Minimizes deletion penalty for data and metadata Flexibility Take and keep as many snapshots as needed Take snapshots of snapshots at any level Create any snapshot tree topology as needed Remove snapshots or their parent volume as needed Automated copy services to refresh the snapshot copies of database with the latest production data XtremIO Management Server Red Hat Enterprise Linux 6.5 EMC backup and recovery solutions XtremIO Management Server (XMS) is a standalone, dedicated, Linux-based server that controls XtremIO system operations. XMS can be either a physical or a virtual server. If the array is disconnected from XMS, it can continue to operate, but cannot be configured or monitored. Red Hat Enterprise Linux (RHEL) is a versatile platform for x86 and x86-64 that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. RHEL 6.5 includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage. EMC backup and recovery solutions EMC Avamar and EMC Data Domain deliver the reliable protection needed to accelerate deployment of a virtualized Oracle server. Optimized for virtualized application environments, EMC backup and recovery technology reduces backup times by 90 percent and increases recovery speeds by 30 times for worry-free protection. EMC backup appliances add another layer of assurance with end-to-end verification and self-healing for ensured recovery. 22 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 3: Solution Overview For full technical guidance, refer to EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11g R2 Design and Implementation Guide. This guide describes how to design, size, and implement EMC backup and recovery solutions for the VSPEX Oracle Database 11g or 12c solution on XtremIO. EMC VSPEX Oracle Computing: Oracle Database Virtualization 23

Chapter 4: Choosing a VSPEX Proven Infrastructure Chapter 4 Choosing a VSPEX Proven Infrastructure This chapter presents the following topics: Overview... 25 Step 1: Evaluate the customer use case... 25 Step 2: Design the application architecture... 26 Step 3: Select the right VSPEX Proven Infrastructure... 26 24 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 4: Choosing a VSPEX Proven Infrastructure Overview This chapter describes how to design the VSPEX Oracle Database 11g or 12c solution on XtremIO and select the right VSPEX Proven Infrastructure to meet your requirements. Table 5 outlines the main steps to complete when selecting a VSPEX Proven Infrastructure. Table 5. VSPEX Proven Infrastructure selection steps Step Action 1 Evaluate the customer s Oracle OLTP/DSS workload by using the VSPEX qualification worksheet for virtualized Oracle databases. Step 1: Evaluate the customer use case provides details. 2 Determine the required infrastructure, Oracle server resources, and architecture using the VSPEX Sizing Tool. Step 2: Design the application architecture provides details. Note: If the Sizing Tool is not available on the EMC Support website, manually size the application using the guidelines in Appendix A. 3 Select the right VSPEX Proven Infrastructure, based on the recommendations from Step 2. Step 3: Select the right VSPEX Proven Infrastructure provides details. Step 1: Evaluate the customer use case Before you select the infrastructure solution, you must understand the customer s real workload and dataset requirements. To help you better understand the customer s business requirements for the VSPEX infrastructure design, EMC strongly recommends that you use the VSPEX qualification worksheet for virtualized Oracle databases in Appendix A. In this worksheet, we ask some simple questions to help understand and describe the customer s Oracle workload requirements and usage characteristics. Table 6 lists and explains the questions. Table 6. VSPEX qualification worksheet for virtualized Oracle databases guidelines Question Do you have an existing Oracle database that you would like to size for in the environment? How many databases do you want to deploy? What type of workload does user database have? Description Select Yes if the customer already has an Oracle database and understands the characteristics that are going to migrate to VSPEX Private Cloud in the VSPEX environment. Type the number of databases that the customer expects to deploy in the VSPEX environment. OLTP-supported only EMC VSPEX Oracle Computing: Oracle Database Virtualization 25

Chapter 4: Choosing a VSPEX Proven Infrastructure Question What is the size of the user database (GB)? What is the annual growth rate (%)? What is the maximum number of IOPS? (Optional) Do you have snapshot databases? How many snapshots does the user want to create and deploy during normal workloads? (Optional) What is the maximum number of bandwidth of the snapshot databases? Description Type the size of the database that the customer expects to have in the VSPEX environment. Future growth is a key characteristic of the VSPEX solution. This value is the expected annual growth rate of the user database in three years. Type a number that is appropriate for the customer s environment. Understanding the maximum number of IOPS of Oracle databases can help to prevent potential storage performance issues. Work with the customer to estimate the IOPS at peak loads. Select Yes if the customer has snapshot databases to deploy. The snapshot is a key characteristic of XtremIO storage. If the customer can estimate the snapshot number at peak loads in the environment, type that number. Work with the customer to estimate the bandwidth of snapshot databases at peak loads. Step 2: Design the application architecture In this VSPEX Proven Infrastructure solution, we defined a representative customer workload to be sized. After you gather the customer s information and populate the VSPEX qualification worksheet for virtualized Oracle databases, you can use that information to populate the VSPEX Sizing Tool located on the EMC Business Value Portal, or you can use the guidelines in Chapter 6 to manually size the solution. Step 3: Select the right VSPEX Proven Infrastructure The VSPEX program provides many solutions designed to simplify the deployment of a consolidated virtual infrastructure using VMware vsphere and the XtremIO array with Data Protection. After you confirm the application architecture, you can select the right VSPEX Proven infrastructure based on the calculated results. Note: While this guide is intended for Oracle database OLTP/DSS requirements, the Oracle server may not be the only application deployed on the VSPEX Proven Infrastructure. You must carefully consider the requirements for each application that you plan to deploy. If you are uncertain about the best VSPEX Proven Infrastructure to deploy, consult an EMC representative before making the decision. 26 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 4: Choosing a VSPEX Proven Infrastructure Follow the steps in Table 7 when choosing a VSPEX Proven Infrastructure. Table 7. Select the right VSPEX Proven Infrastructure Step Action 1 Use the VSPEX Sizing Tool to determine the total number of required resources for virtual machines and any additional suggested storage layout requirements for the Oracle server. 2 Use the VSPEX Sizing Tool to design the resource requirements for other applications, based on business needs. The VSPEX Sizing Tool calculates the total number of required resources for virtual machines and recommended storage layout requirements for both the Oracle server and the other applications. 3 Discuss with your customers the maximum use of VSPEX Proven Infrastructure that meets their business requirements this is the maximum use for both the Oracle server and other applications. Put the maximum utilization percentage of the VSPEX Proven Infrastructure into the VSPEX Sizing Tool. The tool provides a minimum recommendation for the VSPEX Proven Infrastructure offering. 4 Select your network vendor and server vendor for the recommended VSPEX Proven Infrastructure offering. EMC VSPEX: Choose the Right Path to Your Cloud provides more information. The Customer Sizing Worksheet helps you to assess the customer environment and calculate the sizing requirements of the environment. Appendix A provides a blank Customer Sizing Worksheet that you can print out and use to help size the solution for a customer. EMC VSPEX Oracle Computing: Oracle Database Virtualization 27

Chapter 5: Solution Design Considerations and Best Practices Chapter 5 Solution Design Considerations and Best Practices This chapter presents the following topics: Overview... 29 Designing the network... 29 Designing the server... 31 Designing the storage layout... 32 Designing an Oracle database... 35 EMC Data Protection design considerations... 37 28 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 5: Solution Design Considerations and Best Practices Overview Designing the network This chapter describes the best practices and considerations for designing the VSPEX Oracle Database 11g or 12c solution on XtremIO. For more information on deployment of various components of this solution, refer to the vendor-specific documentation. Overview Network best practices VSPEX solutions define minimum network requirements and provide general guidance on network architecture while allowing the customer to choose any network hardware that meets their requirements. EMC recommends that the network infrastructure meet the following requirements: Redundant network links for the hosts, switches, and storage The optimal number of paths depends on the operating system and server information. To avoid multipathing performance degradation, do not use more than 16 paths per device. Balance the hosts between the Storage Controllers to provide a distributed load across all target ports. Host I/O latency is severely affected by ISL (Inter-Switch Links). If possible, place the storage and server ports on the same physical switch. If this is not possible, do not exceed two ISL hops. When setting up the SAN infrastructure for FC, use a single-target per singleinitiator (1:1) zoning scheme. If the FC switch zone count limit has been reached, it is also possible to use a single-target per multiple-initiator (one: many) zoning scheme. If additional bandwidth is needed, it is important to add capacity at both the storage array and the hypervisor host to meet the requirements. Note: Always have at least two physical network connections that are shared by a logical network to ensure that a single link failure does not affect system availability. Design the network so that the aggregate bandwidth, in the event of a failure, is sufficient to accommodate the full workload. VMware vsphere network best practices Networking in virtual environments requires considerations of traffic segmentation, availability, and throughput in addition to the best practices followed in a physical environment. This solution was designed to efficiently manage multiple networks and redundancy of network adapters on ESXi hosts. The following are key best practices for this solution: Separate infrastructure traffic from virtual machine traffic for security and isolation. EMC VSPEX Oracle Computing: Oracle Database Virtualization 29

Chapter 5: Solution Design Considerations and Best Practices Use the VMXNET3 family of par virtualized network adapters. Aggregate physical network cards for network redundancy and performance for example, use pairs of physical NICs per server/vswitch, and uplink each physical NIC to separate physical switches. For more information on networking with vsphere, refer to the instructions in VMware vsphere Networking. Recommended network design This solution provides guidelines for setting up a redundant, highly available network configuration. It provides an example of best practices and design guidelines that apply to the FC storage network. Figure 4 shows an example of a highly available XtremIO FC network. Figure 4. XtremIO FC network example The example shows redundant network links for each ESXi server, storage array, switch interconnect ports, and switch uplink ports. This configuration provides both redundancy and additional network bandwidth, and is required regardless of whether the network infrastructure for the solution already exists or is deployed with other solution components. 30 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Designing the server Chapter 5: Solution Design Considerations and Best Practices For network implementation for the VSPEX Proven Infrastructure, refer to the EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vsphere and EMC XtremIO Implementation Guide. Overview Server best practices VSPEX solutions are designed to run on a wide variety of server platforms. This section describes the minimum CPU and memory resources required. The customer can use any server platform and configuration that meets or exceeds the minimum requirements. EMC recommends that you follow these best practices: Install one or more EMC-approved FC host bus adapters (HBAs) into a Linux host when using FC with XtremIO. XtremIO supports multipathing using EMC PowerPath on Linux. PowerPath provides array-customized loadable array modules (LAMs) native class support for XtremIO volumes. These LAMs feature optimal failover and load balancing behaviors for the XtremIO volumes. Use identical or at least compatible servers to ensure that they share similar hardware configurations. VSPEX implements hypervisor level HA technologies that might require similar instruction sets on the underlying physical hardware. By implementing VSPEX on identical server units, you can minimize compatibility problems in this area. Use recent revisions of common processor technologies for new deployments. These will perform as well as, or better than, the systems used to validate the solution. EMC also recommends the following best practices when configuring a host for VMware vsphere: Implement the HA features available in the virtualization layer to ensure that the compute layer has sufficient resources to accommodate at least single server failures. This will also allow you to implement minimal-downtime upgrades. Ensure that you monitor the performance at both the virtual machine level and the hypervisor level. For example, with an ESXi, you can use performance monitoring within the Oracle database machine to ensure that the virtual machine or Oracle database performs as expected. Meanwhile, at the hypervisor level, you can use monitoring tools such as esxtop to observe host performance. EMC VSPEX Oracle Computing: Oracle Database Virtualization 31

Chapter 5: Solution Design Considerations and Best Practices Validated server design Table 8 lists the server hardware used in this solution. Table 8. Server hardware Components Configuration Server CPU Processor sockets: 4 Cores per socket: 10 2 vcpus per physical core 4 vcpus per reference virtual machine For a heavy load virtual machine: Maximum of 32 vcpus Memory 192 GB RAM 8 GB vram per reference virtual machine For a heavy load virtual machine: Maximum of 64 GB vram Network 2 x 10 GbE NICs per server 2 x HBAs per server VMware vsphere has a number of advanced features to help optimize performance and resource use. The following sections describe the key features and configurations of virtual CPU and memory management and considerations for using them with this solution. Note: Refer to Test methodology and reference workload for more details about the reference virtual machine designed for this solution. Designing the storage layout Overview This section provides guidelines for configuring the storage layer to provide HA and the expected level of performance. This solution used the FC block protocols, and the storage layout adheres to current best practices. Note: EMC unified storage provides flexible management for a storage infrastructure that supports either FC or iscsi protocol. This solution describes only the use of FC with Oracle. XtremIO X-Brick layout XtremIO is based on a flash-optimized scale-out cluster design that linearly adds capacity and performance to meet storage requirements. Each X-Brick has highly available, fully active/active storage controllers with no single point of failure (SPOF). Additional X-Bricks may be added to an existing system, joined together over a redundant, highly available, and ultra-low latency network backbone. In such a system, performance scales linearly while latency remains consistently low. 32 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 5: Solution Design Considerations and Best Practices In this solution, the following XtremIO-validated disk layout has been created to provide support for a specified number of virtual Oracle database servers at a defined performance level. This solution validated two XtremIO configurations: XtremIO Starter X-Brick includes 13 SSD drives XtremIO X-Brick includes 25 SSD drives Note: The XtremIO storage configuration required for this solution is in addition to the storage required by the VSPEX private cloud that supports the solution s infrastructure services. For more information about the VSPEX private cloud storage pool, refer to the VSPEX Proven Infrastructure Guide listed in Essential reading. XtremIO X-Brick scalability XtremIO storage clusters support a fully distributed, scale-out design that allows linear increases in both capacity and performance to provide infrastructure agility. XtremIO used a building-block approach that can be scaled by adding X-Bricks. With clusters of two or more X-Bricks, XtremIO uses a redundant 40 Gbps quad data rate (QDR) Infiniband network for back-end connectivity among the storage controllers. This ensures a highly available, ultra-low latency network. As a result, as capacity in the array grows, performance also grows with the addition of more storage controllers. Figure 5 shows the different configurations as XtremIO storage scales upward. You can start from one single X-Brick, which is a 6U system. As it scales, you can add a second X-Brick, and then a third and fourth X-Brick. Note: In Figure 5, IOPS* (mixed) is measured with a 4 KB fully random workload that is 50 percent writes and 50 percent reads, while IOPS^ (read) is measured with 4 KB and 100 percent reads. Figure 5. XtremIO scalability EMC VSPEX Oracle Computing: Oracle Database Virtualization 33

Chapter 5: Solution Design Considerations and Best Practices Validated XtremIO server virtualization This solution uses a single X-Brick that is validated with the workload profile described in Table 10 and Table 11. The XtremIO array configuration included: Multiple different-sized volumes for Oracle virtual machines. Some volumes stored Oracle data files, and others stored Oracle redo log files and Oracle and Clusterware Ready Service (CRS) files Multiple initiator groups using the vsphere host FC World Wide Names (WWNs) from the hosts in the indicated vsphere environment XtremIO supports the VAAI primitive, thereby enhancing virtual server performance. Figure 6 shows an example volume configuration in the XtremIO console and the volume mapping for different initiator groups. Figure 6. XtremIO volume configuration and mapping The ESXi servers used in this solution were connected to a single X-Brick with two paths of 8 Gbps FC. VMware vsphere provided host-level storage virtualization to virtualize the physical storage and present the virtualized storage to the virtual machine. The LUNs provisioned from the array were then added as virtual disks presented to virtual machines. VMware uses a virtual SCSI controller to present the virtual disk to the guest OS running inside the virtual machine. A virtual machine stores its OS and other files related to the virtual machine activities in a virtual disk. The virtual disk can be one file or multiple files. The virtual disk resides in either a VMware Virtual Machine File system (VMFS) datastore or raw device mapping (RDM). RDM allows the virtual infrastructure to connect a physical device directly to a virtual machine. 34 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 5: Solution Design Considerations and Best Practices vsphere storage virtualization best practices Multiple changes are required to ensure optimal performance of the XtremIO array when used with vsphere. These changes, which are outlined in full in the XtremIO Storage Array User Guide, include: A recommendation to set the following parameters to their maximum values: Disk.SchedNumReqOutstanding Determines the maximum number of active storage commands (I/Os) allowed at any given time at the VMkernel. The maximum value is 256. Disk.SchedQuantum Determines the maximum number of consecutive sequential I/Os allowed from one virtual machine before switching to another virtual machine (unless this is the only virtual machine on the LUN). The maximum value is 64. Disk.DiskMaxIOSize Determines the maximum I/O request size passed to storage devices. With XtremIO, it is required to change it from 32767 (a default setting of 32 MB) to 4096 (4 MB). VAAI is a vsphere API that offloads vsphere operations such as virtual machine provisioning, storage cloning, and space reclamation to storage arrays that support VAAI. When using vsphere version 5.x, VAAI is enabled by default. Therefore, no further action is required to ensure that VAAI is used with XtremIO storage. For optimal performance, we recommend formatting virtual machines on XtremIO storage, using Thick Provision Eager Zeroed: Logical space is allocated and zeroed on virtual machine provisioning time rather than scattered, with each I/O sent by the virtual machine to the disk (when Thick Provision Lazy Zeroed format is used). Thin provisioning is managed in the XtremIO Storage Array rather than in the ESX host (when Thin Provision format is used) In hosts running a hypervisor, such as VMware ESX, Microsoft Hyper-V, or any clustering software, it is important to ensure that the LUNs of XtremIO volumes are consistent across all hosts in the hypervisor cluster. Inconsistent LUNs may affect operations such as virtual machine online migration or virtual machine power-up. The virtual disk resides in either a VMFS datastore or RDM. VMFS is a cluster file system that provides storage virtualization optimized for virtual machines. It can be deployed over any SCSI-based local or network storage. RDM used a FC protocol and allows a virtual machine direct access to a volume on the physical storage. Designing an Oracle database Overview This section provides guidelines for the most common and important design considerations and best practices to follow for an Oracle database. EMC VSPEX Oracle Computing: Oracle Database Virtualization 35

Chapter 5: Solution Design Considerations and Best Practices In this solution, we created virtualized Oracle OLTP and DSS databases on vsphere. With XtremIO, both random I/O and sequential I/O are treated equally as data is randomized and distributed in a balanced fashion throughout the XtremIO array. Oracle storage layout Oracle Automatic Storage Management (ASM) and Oracle CRS are integrated into the Oracle Grid Infrastructure. In this solution, we used ASM to store the relevant database files, including data files, online redo log files, and CRS files. External redundancy was used for the ASM disk groups, and default settings were used for the remaining ASM disk groups attributes. With XtremIO, all drives are under XDP protection, and data in the array is automatically distributed across the X-Bricks to maintain consistent performance and equivalent flash wear levels. We validated the following database storage design on XtremIO X-Brick. The volume size is standardized based on the data usage shown in Table 9. Table 9. Oracle storage design on XtremIO X-Brick XtremIO volume Purpose Volume size (GB) No. of volumes Oracle ASM disk group name Datafile_vol Logfile_vol Crsfile_vol Oracle ASM disks for data files Oracle ASM disks for redo log files Oracle ASM disks for Cluster Registry and voting disk files 1024 4 +DATA 50 4 +REDO 16 2 +CRS Oracle design considerations The following sections describe the best practices and design considerations for Oracle Database 11g or 12c virtualization. Automatic Shared Memory Management (ASMM) is a standard method of dynamically managing memory in an Oracle database and has been available since Oracle Database 10g. EMC recommends that you implement ASMM to automate the management of the following shared memory structures: DB_CACHE_SIZE SHARED_POOL_SIZE LARGE_POOL_SIZE JAVA_POOL_SIZE STREAMS_POOL_SIZE To implement this feature, set the following initialization parameters: SGA_TARGET set to a nonzero value STATISTICS_LEVEL=TYPICAL (or ALL) 36 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 5: Solution Design Considerations and Best Practices The Linux HugePages feature enables the Linux kernel to manage large pages of memory in addition to the standard 4 KB (on x86 and x86_64) or 16 KB (on IA64) page size. HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If the combined database SGAs are large (more than 8 GB), you need to configure HugePages. Note: Do not use Oracle Automatic Memory Management (AMM) because AMM is incompatible with HugePages. If you want to use HugePages, ensure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set. When we created the REDO disk groups, we set the ASM disk group sector size attribute to 4 KB to maximize the I/O performance on the XtremIO storage. We also set the block size of the online redo log files to 4 KB to match the sector size of the REDO disk group. Other recommended database parameter settings include: Set DISK_ASYNCH_IO= true. The default value for this asynchronous I/O in both Oracle 11g and 12c is true. Set FILESYSTEMIO_OPTIONS=SETALL. This setting enables both direct I/O and asynchronous I/O. With asynchronous I/O, processing continues while the I/O request is submitted and processed. Oracle licensing considerations EMC recommends that you consider the Oracle server licensing models to achieve better cost savings in this solution. The Oracle processor licensing option is based on the interaction of the software with hardware. For Oracle EE, the licensing is based on the number of physical cores that are available to the installed Oracle software. For Oracle SE, the licensing is based on the number of processor sockets that are available to the installed Oracle software. Oracle does not permit the soft partitioning of CPUs as a means to calculate or limit the number of software licenses required for a physical server. Oracle regards VMware vsphere technology as soft partitioning, so in a vsphere environment, you must license all hosts where the Oracle executable files are installed and/or running. To minimize Oracle licensing costs, it is essential to correctly design and accurately size the vsphere ESXi cluster, and determine the placement and movement of virtual machines hosting the Oracle executable files. Refer to Understanding Oracle Certification, Support and Licensing for VMware Environments White Paper for more information. Implementing EMC Data Protection All VSPEX solutions are sized and tested with EMC Data Protection products, including Avamar and Data Domain. If your solution includes EMC backup components, refer to EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11g R2 Design and Implementation Guide for detailed information on implementing these options into your VSPEX solution. EMC VSPEX Oracle Computing: Oracle Database Virtualization 37

Chapter 6: Solution Testing and Validation Chapter 6 Solution Testing and Validation This chapter presents the following topics: Overview... 39 Test methodology and reference workload... 39 OLTP workload testing performance results... 41 XtremIO snapshot testing performance results... 44 Verification methodologies... 46 38 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 6: Solution Testing and Validation Overview This chapter provides a summary of the tests we performed to validate the solution. Our objective was to characterize the response time of the VSPEX Oracle solution and the component subsystems under reasonably different loads. The workloads were representative of the sizing logic for Oracle Database 11g R1 or 12c R1 on Linux with XtremIO storage using the recommended VSPEX configuration. This chapter also provides definitions of the reference workload used to test and validate the architectures in this VSPEX solution. It also describes the verification methodologies used for the hardware, application, and data protection aspects of the solution. Test methodology and reference workload Test methodology Swingbench is an easy-to-use, free Java-based tool used to generate database workloads and perform stress testing using different benchmarks in Oracle database environments. You can download the tool from http://dominicgiles.com/downloads.html. Swingbench provides four separate benchmarks, called Order Entry (OE), Sales History (SH), Calling Circle (CC), and Stress Test (ST). In this solution, the Swingbench OE benchmark was used for OLTP workload testing, and the SH benchmark was used for the DSS workload testing. The OE benchmark is based on the OE schema and is from an industry-standard, traditional OLTP benchmark. The workload uses a read/write ratio of 75:25 and is designed to run continuously and test the performance of a typical OE workload against a small set of tables. The SH benchmark is based on the SH schema and is from an industry-standard DSS benchmark. The workload is query (read) centric and is designed to test the performance of queries against large tables. To ensure that the workload execution did not interfere with database performance, the Swingbench tool was run from a separate virtual machine on a different ESXi server. Reference workload We define a reference workload to represent a unit of measure for quantifying the resources in the solution. By comparing the customer s actual usage to this reference workload, you can extrapolate which reference architecture to choose as the basis for the customer s VSPEX deployment. For this purpose, the reference workload was defined as a single-instance Oracle server the reference virtual server with the OLTP/DSS workload characteristics shown in Table 10 and in Table 11. EMC VSPEX Oracle Computing: Oracle Database Virtualization 39

Chapter 6: Solution Testing and Validation Table 10. Reference virtual Oracle server characteristics (OLTP) Characteristic Value Virtual operating system RHEL 6.5 Database version Oracle 11g R2/12c R1 Virtual processors per virtual server 4 Virtual RAM per virtual server Average storage available for each virtual server 8 GB 100 GB Average IOPS per virtual server at steady state 6800 I/O size 8 KB R/W ratio 75/25 Table 11. Reference virtual Oracle server characteristics (DSS) Characteristic Value Virtual operating system RHEL 6.5 Database version Oracle 11g R2/12c R1 Virtual processors per virtual server 1 Virtual RAM per virtual server Average storage available for each virtual server Average bandwidth per virtual server at steady state I/O size 4 GB 100 GB 400 MB/s 32 KB R/W ratio 100/0 If a customer wants larger virtual machines to support a custom application, they can use the reference workload to calculate the number of equivalent reference virtual machines, to get a total of N reference virtual machines. All of the reference virtual machines can be implemented on the same virtual infrastructure that is supported by a single X-Brick. For example, with OLTP workload, we start from one reference virtual machine with 4 vcpus and 8 GB of memory. As we scale and update the inputs to the VSPEX Sizing Tool, the tool generates a series of recommendations, as listed in Table 12. 40 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Table 12. Example virtual server resource requirements Chapter 6: Solution Testing and Validation Virtual machine type Total reference virtual machines Recommended vcpu Recommended vram (GB) Light load 1 4 8 Moderate load 2 8 16 Heavy load 4 16 32 Extremely heavy load 8 32 64 Note: For any workload greater than the extremely heavy load described in Table 12, contact EMC for validation. The VSPEX Sizing Tool enables you to enter a database configuration from the customer s answers into the qualification worksheet. Refer to the VSPEX Sizing Tool portal for more information. If the Sizing Tool is not available on the EMC Support website, use the sizing instructions provided in Appendix A. OLTP workload test performance results OLTP sizing test results The sizing testing was designed as a set of measurements to determine the saturation point of the solution stack in terms of performance. We performed a reasonable amount of fine-tuning to ensure that the performance achieved was consistent with real-world, optimum performance. For sizing purposes, we ran the performance tests and added the workload to show the scalability of the system as more CPU and memory were added. The test results from different rounds of OLTP testing in various configurations are summarized in Figure 7. Figure 7 illustrates the scalability of the system as workload increased and additional CPU and memory were added. EMC VSPEX Oracle Computing: Oracle Database Virtualization 41

Chapter 6: Solution Testing and Validation Figure 7. Workload comparison for a single-instance Oracle database with different virtual machine configurations The test results are detailed as follows: 1. We started from one reference virtual machine with 4 vcpus and 8 GB of virtual memory. We then added a second, third, and fourth reference virtual machine. All of the reference virtual machines can be implemented on the same virtual infrastructure supported by a single X-Brick. 2. We ran different workloads configured with different vcpu/vram combinations against the databases, and we then measured the performance statistics. We also scaled the maximum number of IOPS for a given performance level as configurations were changed from a light load to an extremely heavy load. Note: For more details about a given performance level, refer to Table 13. 3. For each test case, we observed the maximum CPU and memory utilization on both server and storage sides. 4. We successfully executed an extremely heavy Swingbench user workload with a 32 vcpu virtual Oracle Database 12c Server, with I/O wait event response time (shown in the Oracle AWR report) of less than 3 ms, and approximately 1 ms latency at the array. 5. If a customer implements different virtual machines to support a custom-built application, they can use the reference workload to calculate the number of equivalent reference virtual machines, to get a total of N reference virtual machines. 42 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 6: Solution Testing and Validation Figure 8. Single-instance Oracle database CPU utilization with mixed workloads Figure 8 was generated based on the output of the Oracle AWR report. This report shows the average server CPU utilization with different configurations. As observed, more CPU power was needed for a virtual machine during the peak IOPS phase of industry standard traditional OLTP benchmark testing. The highest CPU utilization was still less than 80 percent, which is the CPU metric threshold that must be met in VSPEX Oracle solutions. The results demonstrated the efficiency of XtremIO storage. In Table 13, all related I/O wait events, such as db file sequential read and db file parallel read, were observed to be well within the acceptable range (the average wait time was 1 to 3 ms). The AWR report analysis indicated that the overall system load profile was as expected. Table 13. Performance observation during an OLTP workload execution with 32 vcpus Performance characteristic CPU 32 Memory (GB) 64 Measurements Reads per second 24541 Writes per second 6262 Wait events db file sequential read (average wait time: 1.78 ms) db file parallel read (average wait time: 2.17 ms) db file parallel write (average wait time: 1.87 ms) EMC VSPEX Oracle Computing: Oracle Database Virtualization 43

Chapter 6: Solution Testing and Validation XtremIO snapshot test performance results XtremIO snapshots can be used to provide testing, development, or analytics copies of production data. Provisioning such copies is an easy and instant process. This test was conducted to validate the performance impact of the XtremIO snapshot on the production database, both during snapshot OLTP workload and DSS workload. For sizing purposes, we ran the performance tests and added the workload to show the throughput scalability of the snapshot database as more CPU and memory were added. Figure 9 shows the performance results for a single-instance Oracle database with and without snapshot with 75 percent random reads and 25 percent random writes. Figure 9. Single-instance Oracle database performance with and without a snapshot during OLTP workload The test results are detailed as follows: 1. We used the XtremIO snapshot feature to create a copy of the production database that could be used to run an OLTP workload as production does. The crash-consistent snapshot was taken, mounted on a separate host, and recovered to simulate a workload with and without a running production workload. The workload was run using an OLTP benchmark with 75 percent random reads and 25 percent random writes using small I/O. 2. Figure 9 shows the IOPS for both the production and snapshot databases. As observed, the workload on the production database provided almost the same performance during a 30-minute period of heavy workload as it did without the snapshot database running. The test proved that there was minimal and acceptable impact on the running production OLTP database even while we were accessing both the primary volumes and the snapshot volumes. 3. Figure 10 shows some important Oracle I/O wait events, illustrating the performance of the production database with and without the snapshot database running on XtremIO storage. The workload on the production 44 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 6: Solution Testing and Validation database together with an OLTP workload on the snapshot database can achieve almost the same level of performance as the production database by itself. All of the I/O wait events remain within3 ms. Figure 10. Response time for a single-instance Oracle database with and without a snapshot during OLTP workload 4. In another use case, we validated the performance impact of an XtremIO snapshot on the production database with the DSS workload running against the snapshot. For the snapshot database, we started from one reference virtual machine with 1 vcpu and 4 GB of virtual memory. We then added a second and third reference virtual machine. Using different vcpu/vram combinations, we measured the performance statistics of both the production database and the snapshot database. The maximum IOPS from the production database was recorded, as well as the maximum throughput of different snapshot configurations. 5. Figure 11 shows that consolidating the DSS workload from the snapshot database to the same XtremIO had a negligible performance impact on the production database workload. Scaling the snapshot database configuration from 1 vcpu to 4 vcpus provided linear scaling of bandwidth without a significant drop in IOPS when running OLTP workload on the production database. Figure 11. Single-instance Oracle database performance EMC VSPEX Oracle Computing: Oracle Database Virtualization 45

Chapter 6: Solution Testing and Validation Verification methodologies EMC recommends that you test this VSPEX Proven Infrastructure before deploying it to a production environment. This confirms that the design achieves the required performance and capacity targets, and also identifies potential bottlenecks before they affect users in a live deployment. This section provides a summary description of the high-level steps that we performed when verifying this solution. Before verifying Oracle Database 11g or 12c performance, make sure you have deployed the Oracle database in your VSPEX Proven Infrastructure based on the EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vsphere and EMC XtremIO Implementation Guide. Table 14 describes the high-level steps that you need to complete before you can implement the Oracle database environment in production. Table 14. High-level steps for application verification Step Description Reference 1 Understand key metrics of the Oracle database environment to achieve the performance and capacity your business requires. 2 Use the VSPEX Sizing Tool to determine the required XtremIO array configuration and the compute and network resources for the VSPEX Proven Infrastructure. 3 Design and build the Oracle database solution as described in this document and the solution's Implementation Guide. Run the tests, analyze the results, and optimize the VSPEX architecture. Understand key metrics EMC VSPEX Sizing Tool VSPEX Implementation Guide Understand key metrics Understanding the Oracle server testing goal helps you to decide which metrics to capture and what thresholds must be met for each metric when running the Oracle server validation tests. Use the VSPEX Sizing Tool Use the VSPEX Sizing Tool to understand the basic metrics and thresholds to meet your customer s business requirements. For more information about using the VSPEX Sizing Tool, refer to VSPEX Sizing Tool for Oracle Database 12c, which is available on the EMC VSPEX website. If the VSPEX Sizing Tool is unavailable or does not yet support this solution, use the XtremIO Sizing Tool to determine the storage requirements for deployments of this solution. Refer to the EMC XtremIO Sizing Tool for more information about XtremIO storage sizing. 46 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Follow the VSPEX Implementation Guide Chapter 6: Solution Testing and Validation After you have designed your VSPEX infrastructure, refer to the EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vsphere and EMC XtremIO Implementation Guide for information on how to implement the solution. For this solution, we ran the tests using an application such as TPC-C to validate the Oracle server performance. We recommend that you do the following: Evaluate the workload and I/O pattern. If it is acceptable and the real workload is similar, you can use the test results as a reference. However, customers need to consider the potential risks. Build a test environment first, and then copy and restore the production database to test the real workload and verify the Oracle server performance if the real application workload types are different from what we validated in our test environment. Sizing guidelines Overview Using the Customer Sizing worksheet This section describes how to use the Customer Sizing worksheet to simplify the sizing calculations, and additional factors you should take into consideration when choosing resources. To choose the appropriate reference architecture for a customer environment, you need to determine the resource requirements of the environment and then convert these requirements to an equivalent number of reference virtual Oracle servers that have the characteristics defined in Table 10 and Table 11. Use the VSPEX qualification worksheet for virtualized Oracle databases to collect user requirements. This is a one-page sizing worksheet that you can find in Appendix A of this guide. Table 15 shows a sample VSPEX qualification worksheet for Oracle Database 12c. Table 15. Qualification worksheet example Question Do you have an existing Oracle server database that you would like to size for in the environment? Answer Yes How many databases do you want to deploy? 1 What type of workload does the user database have? OLTP What is the read/write ratio of the workload? 75/25 What is the size of the user database (GB)? 1024 What is the annual growth rate (%)? 10 What is the maximum number of IOPS? 20000 (Optional) Do you have snapshot database(s)? Yes EMC VSPEX Oracle Computing: Oracle Database Virtualization 47

Chapter 6: Solution Testing and Validation Question Answer (Optional) How many snapshot database(s) you want to deploy? 1 (Optional) What is the maximum throughput of the snapshot database(s)? 1000 MB/s Follow these steps to complete the Customer Sizing Worksheet: 1. Identify the number and size of the database or databases planned for migrating or planning on the XtremIO environment and the workload type of each database. 2. For a different workload type, identify the performance metric required IOPS or bandwidth. 3. Identify the annual growth rate and determine whether snapshots will be taken during peak loads. Determining the server resource requirements Based on the server resources validated in this solution, we determined the compute resource requirements in terms of vcpus, vram (GB), and storage performance (IOPS or bandwidth). Table 16 shows the compute resources validated with a typical OLTP workload in this solution. Round all values up to the closest whole number. Table 16. Compute resources validated with a typical OLTP workload Virtual machine type Total reference virtual machines Recommended vcpu Recommended vram (GB) Maximum IOPS supported Light load 1 4 8 6800 Moderate load 2 8 16 13200 Heavy load 4 16 32 20400 Extremely heavy load 8 32 64 30800 We recommend defining a virtual machine as the equivalent of a number of combined reference virtual machines. Otherwise, the virtual machine might not exactly match the specifications that customers need to support. Continue to provision virtual machines with more resources until they are sufficient to host the compute resources that a customer requires. In this case, the necessary virtual machine resources are shown in Table 17. 48 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 6: Solution Testing and Validation Table 17. Required virtual machine resources example (for OLTP) Oracle Server vcpus vram (GB) OS volume capacity (GB) Supported IOPS Resource requirement 16 32 100 20000 To size the compute resources for the snapshot database, we used the validated performance numbers provided in Table 18. The necessary virtual machine resources for snapshot virtual machines are shown in Table 19. Table 18. Compute resources validated with a typical DSS workload Virtual machine type Total reference virtual machines Recommended vcpu Recommended vram (GB) Maximum bandwidth supported (MB/s) Small 1 1 4 400 Medium 2 2 4 1000 Large 4 4 8 1500 Note: We recommend at least 4 GB of memory for Oracle 12c Database to run properly. Table 19. Required virtual machine resources example (for DSS) Oracle Server vcpus vram (GB) OS volume capacity (GB) Supported bandwidth (MB/s) Resource requirement 2 4 100 1000 Note: All performance data contained in this solution was obtained in a rigorously controlled environment. Results obtained in other configurations may vary significantly. Determining the storage resource requirements The storage capacity requirements for virtual Oracle servers can vary widely depending on the size of applications used and the customer datatype. EMC recommends using the XtremIO Sizing Tool to determine the storage resource requirements for deployments of this solution. Gather the values for the required input fields and the appropriate optional input fields for all the databases that will run on XtremIO. The required input fields are database size (GB), IOPS or throughput (MB/s), read block size (KB), and writes block size (KB). Leave the tool's default values for optional input fields if the values are unknown. The XtremIO Sizing Tool determines the recommended type and number of XtremIO X- Bricks for the given workload. Visit the EMC XtremIO Sizing Tool for more information about XtremIO storage sizing. EMC VSPEX Oracle Computing: Oracle Database Virtualization 49

Chapter 6: Solution Testing and Validation Note: Contact EMC for validation of any workload beyond the extremely heavy load and for any workloads that require sub-millisecond latencies enabled by XtremIO. 50 EMC VSPEX Oracle Computing: Oracle Database Virtualization

Chapter 7: Reference Documentation Chapter 7 Reference Documentation This chapter presents the following topics: EMC documentation... 52 Other documentation... 52 EMC VSPEX Oracle Computing: Oracle Database Virtualization 51

Chapter 7: Reference Documentation EMC documentation Other documentation The following documents, available from the EMC Online Support or EMC.com websites, provide additional information. If you do not have access to a document, contact your EMC representative. EMC VSPEX Proven Infrastructure for Virtualized Oracle Data Protection for EMC VSPEX Proven Infrastructure White Paper EMC XtremIO Storage Array User Guide EMC XtremIO Storage Array Host Configuration Guide Oracle documentation VMware documentation The following Oracle documents are relevant to this solution: Oracle Edition Comparisons Oracle Software Investment Guide Oracle Database Licensing Oracle Processor Core Factor Table The following VMware documents are relevant to this solution: Understanding Oracle Certification Support and Licensing for VMware Environments White Paper Oracle Databases on VMware Best Practices Guide Performance Best Practices for VMware vsphere 5.5 52 EMC VSPEX Oracle Computing: Oracle Database Virtualization

EMC Confidential Appendix A: Qualification Worksheet Appendix A Qualification Worksheet This appendix presents the following topics: VSPEX qualification worksheet for virtualized Oracle databases... 54 Gather the information from the customer s Oracle databases example... 54 EMC VSPEX Oracle Computing: Oracle Database Virtualization 53

Appendix A: Qualification Worksheet EMC Confidential VSPEX qualification worksheet for virtualized Oracle databases Before sizing the VSPEX solution, gather the information about the customer s Oracle databases using the qualification worksheet in Table 20. This worksheet is appropriate for qualifying multiple databases. Table 20. VSPEX qualification worksheet for virtualized Oracle databases Question Do you have an existing Oracle server database that you would like to size in the environment? Answer Yes or No How many databases do you want to deploy? What type of workloads does the user database have? OLTP What is the read/write ratio of the workload? What is the size of the user database (GB)? What is the annual growth rate (%)? What is the maximum number of IOPS you require? (Optional) Do you have snapshot database(s)? (Optional) How many snapshot database(s) you want to deploy? (Optional) What is the maximum number of bandwidth of the snapshot database(s)? Gather the information from the customer s Oracle databases example The Oracle AWR provides more of the information required to populate the qualification worksheet for each Oracle database. The AWR provides key statistics on database performance, load, and internal and external resources. You can access this data using standard Oracle-supplied scripts, and obtain the remaining information from the customer, or by using the simple queries in this appendix. Database memory settings Use the init.ora Parameters section of the AWR report to calculate the SGA and PGA values, as shown in Figure 12. 54 EMC VSPEX Oracle Computing: Oracle Database Virtualization

EMC Confidential Appendix A: Qualification Worksheet Figure 12. init.ora Parameters from the AWR Report Find the number of concurrent users Type the following SQL query to confirm the maximum number of users who can concurrently connect to the database: SQL> select SESSIONS_CURRENT, SESSIONS_HIGHWATER from v$license; SESSIONS_CURRENT SESSIONS_HIGHWATER ----------------------------- ---------------------------------- 5 249 1 row selected. Calculate database size Use the data and temporary file sizes to populate the DB Size (MB) column and calculate the total as follows: SQL> select ltrim(to_char(sum(bytes)/(1024*1024))) as Total size (M) from ( select sum(bytes) as bytes from v$datafile union select bytes from v$tempfile); Total size (M) ---------------------------------------- 256000 1 row selected. Find the datafile IOPS and change rate for the redo logs You can find the READ IOPS, WRITE IOPS, and Change Rate (MB/s) values in the IOStat by Function summary section of the AWR report. Figure 13 shows these values. EMC VSPEX Oracle Computing: Oracle Database Virtualization 55

Appendix A: Qualification Worksheet EMC Confidential Figure 13. IOStat by function summary from the AWR Report Calculate user I/O time and commit time The following Oracle wait events, shown in Figure 14, provide key response time statistics for the Oracle database: Use db file sequential read to populate the User I/O column. Oracle recommends that this value be less than 20 ms. Use log file sync to populate the Commit column. Oracle recommends that this value be less than 15 ms. Figure 14. Foreground Wait Event from the AWR report Refer to How to tell if the IO of the Database is Slow (ID 1275596.1) for a list of the typical I/O response times. 56 EMC VSPEX Oracle Computing: Oracle Database Virtualization