IMPLEMENTATION GUIDE EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP EMC VSPEX Abstract This describes the high-level steps and best practices required to implement the EMC VSPEX Proven Infrastructure for Virtualized Oracle Databases on a VSPEX Private Cloud with VMware vsphere enabled by EMC Next- Generation VNX and E MC Backup. It describes implementation for the FAST Cache and FAST VP features and explains their usage in VNX storage systems. Usage guidelines and major customer benefits are also included. October 2013
Copyright 2013 EMC Corporation. All rights reserved. Published in the USA. Published October 2013 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC VSPEX for Virtualized Oracle Database 11g OLTP Enabled by EMC Next- Generation VNX Part Number H12064.1 2 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Contents Contents Chapter 1 Introduction 9 Purpose of this guide... 10 Business value... 10 Scope... 11 Audience... 11 Terminology... 12 Chapter 2 Before You Start 13 Overview... 14 Pre-deployment tasks... 14 Deployment workflow... 14 Deployment prerequisites... 15 Planning and sizing Oracle Database 11g R2... 17 Essential Reading... 17 Design guide... 17 Solution overviews... 17 VSPEX Proven Infrastructure... 17 VMware documentation... 17 Backup and recovery... 17 Chapter 3 Solution Overview 19 Overview... 20 Solution architecture... 20 Key components... 21 Introduction... 21 EMC VSPEX... 22 Oracle Database 11g... 24 VMware vsphere 5.1... 25 VMware vsphere HA... 25 VMware vsphere Distributed Resource Scheduler... 25 VMware vsphere PowerCLI... 25 EMC Next-Generation VNX... 25 VNX performance... 27 Virtualization management... 28 Red Hat Enterprise Linux 6.3... 30 EMC VSPEX For Virtualized Oracle Database 11g OLTP 3
Contents EMC backup and recovery solutions... 30 Chapter 4 Solution Implementation 31 Overview... 32 Setting up the physical components... 32 Implementing the network... 33 Implementing the storage array... 34 Overview... 34 Setting up initial VNX configuration... 35 Provisioning storage for VMFS example... 35 Provisioning storage for NFS example... 36 Configuring FAST Cache (optional)... 40 Configuring FAST VP (optional)... 41 Example storage layout... 43 Implementing ESXi and vcenter... 44 Preparing the Oracle Database 11g R2 virtual machine template... 45 Overview... 45 Creating the Oracle Database 11g R2 virtual machine... 45 Completing the Oracle install pre-requisites... 46 Installing the Oracle Database 11g R2 binaries and patches... 48 Preparing the Oracle Database 11g R2 virtual machine template... 50 Customizing the VMware template... 50 Deploying the Oracle Database 11g R2 virtual machine... 51 Overview... 51 Deploying new virtual machines... 51 Cloning the Oracle binaries ona new virtual machine... 52 Determining the database prerequisites... 53 Configuring the database file layout for NFS... 54 Configuring the database file layout for Block... 56 Configuring the initialization parameters... 57 Enabling the HugePages setting... 58 Creating the database using DBCA... 58 Performing the database post creation activities... 59 Implementing backup and recovery... 61 Overview... 61 Chapter 5 Solution Verification 63 Verifying the baseline infrastructure... 64 Overview... 64 Verifying the ESXi functionality... 64 4 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Contents Verifying the solution component redundancy... 64 Verifying the Oracle Database configuration... 65 Using the health monitoring tools... 65 Verifying the Oracle Database... 66 Overview... 66 Understanding the key metrics... 66 Reviewing the Oracle dataset... 68 Backup and recovery verification methodology... 68 Chapter 6 Reference Documentation 69 EMC documentation... 70 EMC... 70 Oracle... 70 VMware... 70 Other documentation... 71 Links... 71 Appendix A Configuration Worksheet 72 Pre-configuration worksheet for VSPEX... 73 Appendix B: Example Virtual Disk Layout 77 Example Virtual Disk Layout of Oracle Database on the VMFS Datastore... 78 EMC VSPEX For Virtualized Oracle Database 11g OLTP 5
Figures Figures Figure 1. Architecture of the validated infrastructure... 21 Figure 2. VSPEX Proven Infrastructure... 22 Figure 3. Next-Generation VNX with multicore optimization... 27 Figure 4. Active/active processors increase performance, resiliency, and efficiency... 28 Figure 5. New Unisphere Management Suite... 29 Figure 6. Redundant 10 GbE network configuration... 34 Figure 7. Redundant Fibre Channel storage network... 34 Figure 8. Oracle Database 11g R2 storage elements... 35 Figure 9. Example storage layout of Oracle database on VMware... 36 Figure 10. Configuring EMC Unified Storage Management in vsphere Client... 37 Figure 11. Configuring NFS Storage on VNX for Oracle VMs... 37 Figure 12. Configuring VNX storage pools... 38 Figure 13. Unisphere Create File System Wizard... 38 Figure 14. EMC Unisphere Create NFS Export Wizard... 39 Figure 15. Enabling FAST Cache in the Storage Pool Properties dialog... 40 Figure 16. FAST VP creating storage pool... 41 Figure 17. Tier status before data movement... 42 Figure 18. FAST VP in a steady state... 42 Figure 19. Example virtualized oracle storage layout for VSPEX... 43 Figure 20. Oracle Universal Installer Install choose database software only... 48 Figure 21. Enable Oracle Direct NFS client... 49 Figure 22. Example of script root_clone.sh... 49 Figure 23. Example of script clonehome.sh... 49 Figure 24. Virtual machine template example... 50 Figure 25. vsphere custom specification example... 50 Figure 26. Example PowerCLI script to deploy a virtual machine from template... 52 Figure 27. Example PowerCLI script to clone and reattach the ORACLE_HOME... 53 Figure 28. Oracle Net Configuration Assistant utility... 53 Figure 29. Example /etc/mtab NFS mount options... 54 Figure 30. Oranfstab example... 55 Figure 31. Example of /etc/fstab... 57 Figure 32. Oracle Database Configuration Assistant setup screen... 59 Figure 33. Oracle startup/shutdown script /etc/init.d/dbora... 60 Figure 34. Swingbench interface load panel screenshot... 67 Figure 35. Swingbench order entry screen for workload simulation... 67 6 EMC VSPEX For Virtualized Oracle Database 11g OLTP Enabled By EMC Next-Generation VNX and EMC Backup
Figures EMC VSPEX For Virtualized Oracle Database 11g OLTP 7
Tables Tables Table 1. Terminology... 12 Table 2. Tasks for pre-deployment... 14 Table 3. Virtualized oracle deployment workflow for VSPEX... 14 Table 4. Deployment prerequisites checklist... 16 Table 5. Reference virtual machine characteristics... 23 Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine... 23 Table 7. Tasks for physical setup... 32 Table 8. Tasks for switch and network configuration... 33 Table 9. Tasks for storage configuration... 34 Table 10. Steps for installing and configuring ESXi and vcenter... 44 Table 11. Virtual machine template specification... 45 Table 12. Oracle install prerequisites... 46 Table 13. RHEL 6 RPM packages for Oracle Database 11g R2... 47 Table 14. Installing the Oracle Database binaries and patches... 48 Table 15. Oracle Database 11g R2 reference virtual machine model... 51 Table 16. Database file layout for NFS... 54 Table 17. Tasks for verifying the VSPEX Proven Infrastructure installation... 64 Table 18. Tools for monitoring the solution... 65 Table 19. General operations... 66 Table 20. Schema tables and indexes... 68 Table 21. Common server information... 73 Table 22. ESXi server information... 73 Table 23. Array information... 74 Table 24. Network infrastructure information... 74 Table 25. VLAN information... 74 Table 26. Service accounts... 75 Table 27. Example Virtual Disk Layout... 78 8 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 1: Introduction Chapter 1 Introduction This chapter presents the following topics: Purpose of this guide... 10 Business value... 10 Scope... 11 Audience... 11 Terminology... 12 EMC VSPEX For Virtualized Oracle Database 11g OLTP 9
Chapter 1: Introduction Purpose of this guide Business value EMC VSPEX Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk. VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure. The VSPEX for virtualized Oracle infrastructure provides customers with a modern system, capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vsphere to provide virtualization, with EMC Next-Generation VNX providing storage, and EMC Avamar and EMC Data Domain for backup. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment. This describes how to implement, using best practices, the VSPEX Proven Infrastructure for virtualized Oracle Databases on a VSPEX Private Cloud with VMware vsphere. Database management system software continues to be the dominant software used to manage data in nearly all commercial segments. This growth is expected to continue despite the increasing market share of other data management tools. This growth is expected to accelerate as customers continue to diversify their infrastructures and supporting technologies and drive towards more hardware and software appliances and configurations. This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNX series, EMC backup and recovery systems, and Oracle bring to customers who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues. This VSPEX solution is designed to meet the customer s Oracle database challenges while enabling customers to grow in performance, scalability, reliability, and automation. By consolidating their database applications on VNX, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today. This solution has been sized and proven to: Deploy faster, saving time and effort with Proven solutions Increase performance and scalability out of the box Reduce the customer s backup storage requirements and costs Accommodate backup windows Enable fast disk-based recovery 10 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 1: Introduction Scope This describes the high-level steps required to deploy the VSPEX solution for Oracle RDBMS and Online Transaction Processing (OLTP) systems. This VSPEX application solution is based on the current VSPEX private cloud with VMware solution and has been modified to support Oracle Database requirements. This solution uses the VSPEX standard or reference virtual machine building blocks of vcpu, memory, and storage and I/O configurations, and describes the implementation based on three (small, medium, and large) different database workload thresholds. This assumes that a VSPEX Proven Infrastructure already exists in the customer s environment, which includes VMware vsphere,vnx, and EMC backup and recovery systems. Audience This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes that the VSPEX partners who intend to deploy this solution are: Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures Certified for selling VSPEX Proven Infrastructures Partners implementing this solution should also have the necessary technical training and background to install and configure: VMware vsphere 5.1 Red Hat Enterprises Linux 6.3 Oracle Database 11g R2 or above EMC next-generation backup, which includes EMC Avamar and EMC Data Domain This document provides external references where applicable. EMC recommends that readers are familiar with these documents. For details, refer to the section entitled EMC documentation. EMC VSPEX For Virtualized Oracle Database 11g OLTP 11
Chapter 1: Introduction Terminology Table 1 lists the terminology used in the guide. Table 1. Terminology Term AWR DNFS DNS emlc FAST VP FQDN IOPS FRA NFS NL-SAS OLTP Oracle EE Oracle SE PowerCLI Reference virtual machine SGA Statspack TPS VDM VMDK VMFS Definition Automatic Workload Repository Oracle Direct NFS client Domain name system Enterprise multilevel cell Fully Automated Storage Tiering for Virtual Pools Fully Qualified Domain Name Input/output operations per second Fast Recovery Area (Oracle) Network File System Near-line serial-attached SCSI Online transaction processing Oracle Enterprise Edition Oracle Standard Edition A Windows PowerShell interface to the VMware vsphere and vcloud APIs Represents a unit of measure for a single virtual machine to qualify the compute resources in a VSPEX Proven Infrastructure System global area Oracle database monitoring and reporting utilities Transactions per second Virtual Data Mover VMware virtual machine disk VMware virtual machine file system 12 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 2: Before You Start Chapter 2 Before You Start This chapter presents the following topics: Overview... 14 Pre-deployment tasks... 14 Deployment workflow... 14 Deployment prerequisites... 15 Planning and sizing Oracle Database 11g R2... 17 Essential Reading... 17 EMC VSPEX For Virtualized Oracle Database 11g OLTP 13
Chapter 2: Before You Start Overview Before you layer the Oracle OLTP on a VSPEX private cloud, EMC recommends that you complete the pre-deployment tasks listed in Table 2. Pre-deployment tasks Pre-deployment tasks include procedures that do not directly relate to environment installation and configuration, but whose results are needed during installation. Examples of pre-deployment tasks include the collection of hostnames, IP addresses, VLAN IDs, license keys, installation media, and so on. Perform these tasks before visiting the customer to decrease the time required on site. Table 2. Tasks for pre-deployment Task Description Reference Gather documents Gather tools Gather data Gather the related documents listed in Essential Reading. These are used throughout the text of this document to provide details on setup procedures and deployment best practices for the various components of the solution. Gather the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process. Collect the customer-specific configuration data for networking, naming, and required accounts. Enter this information into the Customer Oracle Database 11g R2 configuration worksheet for reference during the deployment process. Essential Reading Deployment prerequisites Appendix A Configuration Worksheet Deployment workflow EMC recommends that you follow the process flow in Table 3 to design and implement your VSPEX Proven Infrastructure for virtualized Oracle Database 11g R2 solution. Table 3. Virtualized oracle deployment workflow for VSPEX Step Action 1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in Appendix A of the companion EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP Design Guide 14 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 2: Before You Start Step Action 2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 11g R2 solution. For more information about the VSPEX Sizing Tool, refer to the VSPEX Sizing Tool on the EMC Business Value Portal. Note You need to register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application using the sizing guidelines in Appendix B of the companion Design Guide. 3 To determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 11g, refer to the companion Design Guide. Note Ensure that all application requirements are considered, not just this application. 4 To select and order the right solution, refer to the appropriate VSPEX Proven Infrastructure document in Essential Reading. 5 To deploy and test your VSPEX solution, refer to this guide. Deployment prerequisites Table 4 itemizes the hardware and software requirements to configure the solution. For additional information, refer to the hardware and software tables in the relevant document in the Essential Reading section. EMC VSPEX For Virtualized Oracle Database 11g OLTP 15
Chapter 2: Before You Start Requirement Table 4. Deployment prerequisites checklist Description Version Reference notes Hardware Physical servers: sufficient physical server capacity to host the required number of virtual machines as recommended by the companion Design Guide and VSPEX Sizing Tool. VMware vsphere 5.1 servers to the host virtual infrastructure servers. This requirement may be covered in the existing infrastructure. Networking: switch port capacity and capabilities as required by the virtual server infrastructure. EMC VNX: multiprotocol storage array with the required disk layout. EMC Backup: Avamar 6.1, Data Domain Reference Architecture: EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1000 Virtual Machines EMC Avamar 6.1 hardware and licenses EMC Data Domain hardware and licenses VNX OE for file VNX OE for block Release 8.0.10000.3 4653 Release 05.33.000.3.653 Software EMC VSI for VMware vsphere: Unified Storage Management 5.4 EMC VSI for VMware vsphere: Storage Viewer 5.4 VMware ESXi installation media 5.1 VMware vcenter Server installation media 5.1 EMC online support EMC Avamar Red Hat Enterprise Linux 6.3 6.1 in server and client versions Red Hat Enterprise Linux 6.3 media www.redhat.com VMware vsphere license keys. This may be covered in the VSPEX Proven Infrastructure. 5.1 www.vmware.com Licenses Red Hat Enterprise Linux 6.3 www.redhat.com Oracle Database 11g R2 11.2.0.3.6 www.oracle.com 16 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 2: Before You Start Planning and sizing Oracle Database 11g R2 Essential Reading To plan and size your Oracle Database 11g R2 deployment on the VSPEX infrastructure, you should follow the recommendations in the companion Design Guide. Size the storage accordingly to information obtained from the VSPEX for Virtualized Oracle 11g Qualification worksheet as detailed in Appendix A and B of the companion Design Guide. Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space in the EMC Community Network or from EMC.com or the VSPEX Proven Infrastructure partner portal. Design guide Solution overviews VSPEX Proven Infrastructure Design Guide: EMC VSPEX for Virtualized Oracle Database 11g OLTP EMC VSPEX Server Virtualization for Midmarket Businesses EMC VSPEX Server Virtualization for Small and Medium Businesses Reference Architecture: EMC VSPEX Private Cloud: VMware vsphere for up to 1,000 Virtual Machines White Paper: EMC Avamar Backup for Oracle Environments White Paper: EMC Avamar Backup with Data Domain VMware documentation Backup and recovery VMware VSphere Product Documentation Refer to the following backup and recovery papers: White Paper: EMC Avamar Backup for Oracle Environments White Paper: EMC Avamar Backup with Data Domain White Paper: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and i EMC VSPEX For Virtualized Oracle Database 11g OLTP 17
Chapter 2: Before You Start 18 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 3: Solution Overview Chapter 3 Solution Overview This chapter presents the following topics: Overview... 20 Solution architecture... 20 Key components... 21 EMC VSPEX For Virtualized Oracle Database 11g OLTP 19
Chapter 3: Solution Overview Overview Solution architecture This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 11g and the key technologies used in this solution. The solution described in this includes servers, storage, network components, and Oracle Database 11g components. The solution enables customers to quickly and consistently deploy a virtualized Oracle Database 11g in the VSPEX Proven Infrastructure. The reference architecture will consume the reference virtual machine resources, based on the sizing guidance in the VSPEX Proven Infrastructure, and combine with additional storage for Oracle Database 11g application data. This can help EMC personnel and qualified EMC VSPEX Partners to deploy a simple, effective, and flexible Oracle Database 11g solution on a VSPEX Proven Infrastructure for their customers. Figure 1 shows the architecture that characterizes the infrastructure validated for an Oracle Database 11g overlay on a VSPEX infrastructure. To validate this solution, we 1 : Deployed all Oracle Database 11g servers as virtual machines on VMware vsphere 5.1. Used the VSPEX sizing tool for Oracle Database 11g to determine the number of, and the detailed compute resources for, each Oracle Database 11g database. Figure 1 displays an example with three Oracle sizing options (small, medium, and large). Use the sizing tools provided with this solution to size your customer s environment and choose the options that best suit your customer. Determined the recommended storage layout for Oracle Database 11g and the virtual infrastructure pool in the VNX series storage arrays (using the VSPEX sizing tool). Note: The minimum Oracle version for this solution is 11.2.0.3. We refer to this as 11gR2 throughout this document. 1 In this paper, "we" refers to the EMC Solutions engineering team that validated the solution. 20 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 3: Solution Overview Figure 1. Architecture of the validated infrastructure Key components Introduction This section provides an overview of the key technologies used in this solution: EMC VSPEX Oracle Database 11g VMware vsphere 5.1 VMware vsphere HA vsphere Distributed Resources Scheduler VMware vsphere PowerCLI EMC Next-Generation VNX Red Hat Enterprise Linux 6.3 EMC Avamar EMC Data Domain EMC VSPEX For Virtualized Oracle Database 11g OLTP 21
Chapter 3: Solution Overview EMC VSPEX EMC has joined forces with the industry s leading providers of IT infrastructure to create a complete virtualization solution that accelerates the deployment of private cloud technologies. Built with best-of-breed technologies, VSPEX enables faster deployment, more simplicity, greater choice, higher efficiency, and lower risk. VSPEX Proven Infrastructure, as shown in Figure 2, is a modular, virtualized system validated by EMC and delivered by EMC partners. VSPEX includes a virtualization layer, server, network, and storage, designed by EMC to deliver reliable and predictable performance. Figure 2. VSPEX Proven Infrastructure VSPEX provides the flexibility to choose the best network, server, and virtualization technologies that fit a customer s environment to create a complete virtualization solution. VSPEX provides a virtual infrastructure for customers looking to gain the simplicity of a truly converged infrastructure, while gaining flexibility in individual components of the stack. VSPEX solutions, proven by EMC, are packaged and sold exclusively by EMC channel partners. VSPEX provides channel partners with more opportunity, a faster sales cycle, and end-to-end enablement. By working even more closely together, EMC and its channel partners can now deliver an infrastructure that accelerates the journey to the cloud for even more customers. 22 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Reference virtual machine Chapter 3: Solution Overview To simplify the virtual infrastructure discussion, the VSPEX solution has defined a typical customer workload (described in this section) as a reference virtual machine. For VSPEX solutions, we define the reference virtual machine as a measure unit of a single virtual machine to qualify the compute resources in the VSPEX virtual infrastructure. Table 5 lists the characteristics of this virtual machine. Table 5. Reference virtual machine characteristics Characteristic Virtual processors per virtual machine 1 Value RAM per virtual machine Available storage capacity per virtual machine I/O operations per second (IOPS) per virtual machine I/O pattern 2 GB 100 GB 25 Random I/O read/write ratio 2:1 VSPEX for virtualized Oracle sizing model Scale-up testing formed part of the validation process. We used a standard computesizing model for Oracle, which simplified and standardized the validation testing. It also enabled us to identify the configuration required to run a TCP-C like OLTP database workload with a 60:40 read/write ratio, yielding acceptable response times. Table 6 shows how we mapped the Oracle sizing model to the VSPEX reference virtual machine. Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine Oracle model Small virtual machine for up to 150 users Medium virtual machine for up to 250 users Resources Compute requirements: 2 vcpu 8 GB memory Storage requirements (OS & Oracle binaries): 100 GB 25 IOPS Compute requirements: 4 vcpu 16 GB of memory Storage requirements (OS & Oracle binaries): 100 GB 25 IOPS Equivalent reference virtual machine 4 8 EMC VSPEX For Virtualized Oracle Database 11g OLTP 23
Chapter 3: Solution Overview Oracle model Large virtual machine for more than 250 Users Resources Compute requirements: 8 vcpu 32 GB of memory Storage requirements (OS & Oracle binaries): 100 GB 25 IOPS Equivalent reference virtual machine 16 We calculated the database storage I/O thresholds and capacity separately from those required by the VSPEX reference virtual machine. Oracle Database 11g Oracle Database 11g is available in a variety of editions tailored to meet the business and IT needs of an organization. In this solution we will be considering Oracle Database 11g Release 2 Standard Edition (SE) Oracle Database 11g Release 2 Enterprise Edition (EE) Oracle Database 11g SE is an affordable, full-featured data management solution that is ideal for all companies. It is available on single or clustered servers and can be licensed on a maximum capacity of four processor sockets, regardless of core count. The SE license includes Oracle Real Application Clusters (RAC) as a standard feature with no additional cost. Oracle Database 11g EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It supports advanced features, either included or as extra-cost options, that are not available with Oracle Database 11g SE. These include security features such as Virtual Private Database and data warehousing options such as partitioning and advanced analytics. Oracle Database 11g Release 2 EE extends the processorlicensing model for multi-core processors and is priced using the following formula: (number of Processors) x (number of cores) x (Oracle Processor Core Factor) For example, two 10-core Intel Xeon Processor E7-2870s (with an Oracle Processor Core Factor of 0.5) are licensed as follows Oracle Database 11g Release 2 SE: 2 processor socket SE licenses Oracle Database 11g Release 2 EE: 2 x 10 x 0.5 = 10 EE licenses The Oracle Database 11g R2 edition can affect the licensing cost and the size and number of VMware ESXi clusters you can configure. This affects how you place and manage the virtual machines. Refer to the document entitled Design Guide for EMC VSPEX for Virtualized Oracle Database 11g OLTP, in the DRS Host Affinity and Oracle processor licensing section, for more information on virtualization and Oracle processor licensing. 24 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 3: Solution Overview VMware vsphere 5.1 VMware vsphere HA VMware vsphere 5.1 abstracts applications and information from the complexity of underlying infrastructure through comprehensive virtualization of server, storage, and networking hardware. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications just like physical computers. This virtualization of hardware resources enables efficiencies through consolidation of multiple applications on fewer physical servers. VMware vsphere High Availability (HA) provides easy-to-use, cost effective high availability for applications running in virtual machines. In the event of a physical server failure, affected virtual machines automatically restart on other production servers with spare capacity. HA enables you to create a cluster out of multiple ESXi servers, enabling you to protect virtual machines. If one of the hosts in the cluster fails, the impacted virtual machines automatically restart on other ESXi hosts within that same VMware vsphere cluster. VMware vsphere Distributed Resource Scheduler VMware vsphere PowerCLI EMC Next- Generation VNX VMware vsphere Distributed Resource Scheduler (DRS) is an infrastructure service run by VMware vcenter Server (vcenter). DRS aggregates ESXi host resources into clusters and automatically distributes these resources to virtual machines by monitoring utilization and continuously optimizing virtual machine distribution across ESXi hosts. DRS can also use vmotion and Storage vmotion to ensure that the virtual machines have access by rebalancing resource capacity to make room for larger virtual machines. VMware recommends enabling DRS to achieve higher consolidation ratios. VMware vsphere PowerCLI provides a Windows PowerShell interface for the users of vsphere 5.1 and above and VMware Infrastructure 4.x and above. VMware vsphere PowerCLI is a powerful command-line tool that lets you automate all aspects of vsphere management, including network, storage, VM, guest OS and more. PowerCLI is distributed as a Windows PowerShell snap-in, and includes 330 PowerShell cmdlets for managing and automating vsphere and vcloud, along with documentation and samples. The EMC VNX flash-optimized unified storage platform delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. Ideal for mixed workloads in physical or virtual environments, VNX combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today s virtualized application environments. VNX includes many features and enhancements designed and built upon the first generation s success. These features and enhancements include: More capacity with multicore optimization with Multicore Cache, Multicore RAID, and Multicore FAST Cache (MCx) Greater efficiency with a Flash-optimized hybrid array Better protection by increasing application availability with active/active EMC VSPEX For Virtualized Oracle Database 11g OLTP 25
Chapter 3: Solution Overview Easier administration and deployment by increasing productivity with new Unisphere Management Suite VSPEX is built with the next generation of VNX to deliver even greater efficiency, performance, and scale than ever before. Flash-optimized hybrid array VNX is a flash-optimized hybrid array that provides automated tiering to deliver the best performance to your critical data, while intelligently moving less frequently accessed data to lower-cost disks. In this hybrid approach, a small percentage of flash drives in the overall system can provide a high percentage of the overall IOPS. Flash-optimized VNX takes full advantage of the low latency of flash to deliver cost-saving optimization and high performance scalability. The EMC Fully Automated Storage Tiering Suite (FAST Cache and FAST VP) tiers both block and file data across heterogeneous drives and boosts the most active data to the flash, ensuring that customers never have to make concessions for cost or performance. Data is generally accessed most frequently at the time it is created, therefore new data is first stored on flash drives to provide the best performance and latency. As that data ages and becomes less active over time, FAST VP tiers the data from highperformance to high-capacity drives automatically, based on customer-defined policies. This functionality has been enhanced with four times better granularity and with new FAST VP solid-state disks (SSDs) based on enterprise multi-level cell (emlc) technology to lower the cost per gigabyte. FAST Cache dynamically absorbs unpredicted spikes in system workloads. All VSPEX use cases benefit from the increased efficiency. VSPEX Proven Infrastructures deliver private cloud, end-user computing, and virtualized application solutions. With VNX, customers can realize an even greater return on their investment. VNX provides out-of-band, block-based deduplication that can dramatically lower the costs of the Flash tier. VNX Intel MCx Code Path Optimization The advent of Flash technology has been a catalyst in totally changing the requirements of midrange storage systems. EMC redesigned the midrange storage platform to efficiently optimize multicore CPUs to provide the highest performing storage system at the lowest cost in the market. MCx distributes all VNX data services across all cores up to 32, as shown in Figure 3. The VNX series with MCx has dramatically improved the file performance for transactional applications like databases or virtual machines over network-attached storage (NAS). 26 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 3: Solution Overview Figure 3. Next-Generation VNX with multicore optimization Multicore Cache The cache is the most valuable asset in the storage subsystem; its efficient use is key to the overall efficiency of the platform in handling variable and changing workloads. The cache engine has been modularized to take advantage of all the cores available in the system. Multicore RAID Another important part of the MCx redesign is the handling of I/O to the permanent back-end storage hard disk drives (HDDs) and SSDs. Greatly increased performance improvements in VNX come from the modularization of the back-end data management processing, which enables MCx to seamlessly scale across all processors. VNX performance VNX storage, enabled with the MCx architecture, is optimized for FLASH 1 st and provides unprecedented overall performance, optimizing for transaction performance (cost per IOPS), bandwidth performance (cost per GB/s) with low latency, and providing optimal capacity efficiency (cost per GB). VNX provides the following performance improvements: Up to four times more file transactions when compared with dual controller arrays Increased file performance for transactional applications (for example, Microsoft Exchange on VMware over NFS) by up to three times with a 60 percent better response time Up to four times more Oracle and Microsoft SQL Server OLTP transactions Up to six times more virtual machines EMC VSPEX For Virtualized Oracle Database 11g OLTP 27
Chapter 3: Solution Overview Active/active array service processors The new VNX architecture provides active/active array service processors, as shown in Figure 4, which eliminate application timeouts during path failover since both paths are actively serving I/O. Note: The active/active processors are only available for classic LUNs not for pool LUNs. Load balancing is also improved and applications can achieve an up to two times improvement in performance. Active/active for block is ideal for applications that require the highest levels of availability and performance, but do not require tiering or efficiency services like compression, deduplication, or snapshot. With this VNX release, VSPEX customers can use virtual Data Movers (VDMs) and VNX Replicator to perform automated and high-speed file system migrations between systems. This process migrates all snaps and settings automatically, and enables the clients to continue operation during the migration. Figure 4. Active/active processors increase performance, resiliency, and efficiency Virtualization management VMware Virtual Storage Integrator Virtual Storage Integrator (VSI) is a no-charge VMware vcenter plug-in available to all VMware users with EMC storage. VSPEX customers can use VSI to simplify management of virtualized storage. VMware administrators can gain visibility into their VNX storage using the same familiar vcenter interface to which they are accustomed. With VSI, IT administrators can do more work in less time. VSI offers unmatched access control that enables you to efficiently manage and delegate storage tasks with confidence. Perform daily management tasks with up 90 percent fewer clicks and up to 10 times higher productivity. VMware vstorage APIs for Array Integration VMware vstorage APIs for Array Integration (VAAI) offloads VMware storage-related functions from the server to the storage system, enabling more efficient use of server and network resources for increased performance and consolidation. 28 EMC VSPEX For Virtualized Oracle Database 11g OLTP
VMware vstorage APIs for Storage Awareness Chapter 3: Solution Overview VMware vstorage APIs for Storage Awareness (VASA) is a VMware-defined API that displays storage information through vcenter. Integration between VASA technology and VNX makes storage management in a virtualized environment a seamless experience. EMC Storage Integrator EMC Storage Integrator (ESI) is targeted towards the Windows and Application administrator. ESI is easy to use, delivers end-to end monitoring, and is hypervisor agnostic. Administrators can provision in both virtual and physical environments for a Windows platform, and troubleshoot by viewing the topology of an application from the underlying hypervisor to the storage. Unisphere Management Suite EMC Unisphere is the central management platform for the VNX series, providing a single, combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration, automatically discovering virtual machines and ESX servers and providing end-to-end, virtual-to-physical mapping. Unisphere also simplifies configuration of FAST Cache and FAST VP on VNX platforms. The new Unisphere Management Suite extends Unisphere s easy-to-use, interface to include VNX Monitoring and Reporting for validating performance and anticipating capacity requirements. As shown in Figure 5, the suite also includes Unisphere Remote for centrally managing up to thousands of VNX and VNXe systems with new support for XtremSW Cache. Figure 5. New Unisphere Management Suite EMC VSPEX For Virtualized Oracle Database 11g OLTP 29
Chapter 3: Solution Overview Red Hat Enterprise Linux 6.3 EMC backup and recovery solutions Red Hat Enterprise Linux is a versatile platform for x86 and x86-64 that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. Red Hat Enterprise Linux 6.3 includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage. EMC Avamar and EMC Data Domain deliver the protection confidence needed to accelerate deployment of virtualized Oracle. Optimized for virtualized application environments, EMC backup and recovery reduces backup times by 90 percent and increases recovery speeds by 30 times,even offering instant virtual machine access,for worry-free protection. EMC backup also delivers big savings. Our deduplication solutions reduce backup storage by 10 to 30 times, backup management time by 81 percent, and bandwidth by 99 percent for efficient offsite replication, delivering a seven month payback on average. Furthermore, EMC backup offers a solution with Data Domain systems and DD Boost software that allows for full DBA control of Oracle backup, recovery, and replication while the backup team maintains control of the infrastructure. This eliminates the occurrence of protection silos, increasing efficiency and lowering risk. 30 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Chapter 4 Solution Implementation This chapter presents the following topics: Overview... 32 Setting up the physical components... 32 Implementing the network... 33 Implementing the storage array... 34 Implementing ESXi and vcenter... 44 Preparing the Oracle Database 11g R2 virtual machine template... 45 Deploying the Oracle Database 11g R2 virtual machine... 51 Implementing backup and recovery... 61 EMC VSPEX For Virtualized Oracle Database 11g OLTP 31
Chapter 4: Solution Implementation Overview This chapter discusses the steps required to install, configure, and implement the VSPEX Proven Infrastructure environment, which involves: Preparing the components and their physical setup. Implementing the network, including the configuration steps for physical and virtual switches in line with vendor guidelines and The vsphere Networking Guide. Configuring the VNX storage array and optional FAST Cache or FAST VP feature. Building the VMware vsphere environment with ESXi server and vcenter implementation. Preparing a virtual machine template for rapid deployment of Oracle Database 11g R2. Quickly and simply deploying an Oracle Database 11g R2 virtual machine using PowerCLI. Implementing backup and recovery. Setting up the physical components This section includes information about how to prepare the solution s physical components. After you complete the steps listed in Table 7, the new hardware components will be racked, cabled, powered, and ready for network connection. Note If you already have a VSPEX Proven Infrastructure environment, you can skip this section. Table 7. Tasks for physical setup Task Description Reference Prepare network switches Prepare servers Prepare VNX Install switches in the rack and connect them to power. Install the servers in the rack and connect them to power. Install the VNX in the rack and connect it to power. Your vendor s installation guide Your vendor s installation guide EMC VNX System Installation Guide For details of the physical setup, refer to Reference Architecture: EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1,000 Virtual Machines 32 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Implementing the network This section documents requirements for the network infrastructure needed to support this architecture. Table 8 provides a summary of the tasks for switch and network configuration and references for further information. Note: If you already have a VSPEX Proven Infrastructure environment, you can skip this section. Table 8. Tasks for switch and network configuration Task Description Reference Configure network Complete network cabling Configure VLAN Configure VMare vsphere networking Configure storage network and host infrastructure networking as specified in the solution reference architecture. Connect: Switch interconnect ports VNX ports ESXi NIC/FC ports Configure private and public VLANs as required. Create virtual switches for separation of application, management, and storage networks. Refer to the appropriate document in Essential Reading. N/A Vendor s switch configuration guide vsphere Networking Guide For details of network implementation, refer to the appropriate document in Essential Reading. Figure 6 and Figure 7 show the configured vsphere network for this implementation. As highlighted, redundant 10GbE network adapters or FC Host adapters have been dedicated for storage network traffic. EMC VSPEX For Virtualized Oracle Database 11g OLTP 33
Chapter 4: Solution Implementation Figure 6. Redundant 10 GbE network configuration Figure 7. Redundant Fibre Channel storage network Implementing the storage array Overview This section describes how to configure the VNX storage array. In this solution, the VNX provides Network File System (NFS) or Virtual Machine File System (VMFS) data storage for the VMware host. Table 9. Tasks for storage configuration Task Description Reference Set up initial VNX configuration Provision storage Configure the IP address information and other key parameters on the VNX. Create VMFS datastores or create NFS file systems that will be mounted on hosts VNX System Installation Guide VNX Series Configuration Worksheet 34 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Figure 8 depicts the high-level architecture of this solution. Chapter 4: Solution Implementation Figure 8. Oracle Database 11g R2 storage elements Setting up initial VNX configuration Ensure that network interfaces, IP address information, and other key parameters such as DNS and NTP are configured on the VNX before provisioning the storage. For more information on how to configure the VNX platform, refer to the appropriate document in Essential Reading. Provisioning storage for VMFS example Before you provision the storage for a VMFS datastore, follow the recommendations and VSPEX Sizing Tool proposals introduced in the companion Design Guide. Before you start, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1000 Virtual Machines to provision storage for virtual machine operating system on VNX. Figure 9 shows an example of how to provision a VMFS datastore for the Oracle Database 11g R2 data in VNX. For more information about the storage layout recommendations and design, refer to the Design Guide. EMC VSPEX For Virtualized Oracle Database 11g OLTP 35
Chapter 4: Solution Implementation Figure 9. Example storage layout of Oracle database on VMware For more information on how to lay out virtual disks for the Oracle Database 11g R2 data on the VMFS datastore, refer to the example in Table 27 on page 78. Provisioning storage for NFS example Present the NFS filesystem for the virtual machine datastore to all ESXi hosts, using either EMC Unisphere or the VSI plugin for VMware vsphere client. Figure 11 displays how how to mount the NFS file systems to ESXi hosts that are used to store virtual servers on the VNX array using VSI. The EMC Unified Storage Management plugin for VMware vsphere provides an interface for creating and managing NFS file systems for VMware ESXi host directly from the vsphere Client interface. Figure 10 shows the configuration screen accessed through Home >Solutions and applications >EMC from the vsphere Client, which is accessible after running the installer on your management host. 36 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Figure 10. Configuring EMC Unified Storage Management in vsphere Client Unified Storage Management enables the vsphere administrator to mount NFS file systems directly to the ESXi cluster nodes from vsphere Client, as shown in Figure 11. Figure 11. Configuring NFS Storage on VNX for Oracle VMs To configure NFS network settings, storage pools for file, file systems, and NFS exports on the VNX array in EMC Unisphere, perform the following steps: 1. In Unisphere, select the VNX array for this solution. 2. Select Settings Network Settings for File 3. Configure the IP address for network ports used for NFS. For detailed steps refer to EMC Procedure Generator for VNX. 4. Select Storage Storage Configuration EMC VSPEX For Virtualized Oracle Database 11g OLTP 37
Chapter 4: Solution Implementation Figure 12. Configuring VNX storage pools 5. Select the Storage Pools for File tab and create the additional storage pools in the VNX for Oracle Database files, as shown in Figure 12. Refer to Table 27 on page 78 for detailed information. 6. Create the required file systems. Figure 13 shows the Create File System Wizard from Unisphere. Figure 13. Unisphere Create File System Wizard 38 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation 7. Create the NFS exports for the database components that the Oracle NFS client will mount on the database virtual machines. Figure 14 shows the Unisphere Create NFS Export Wizard. Root and Access permissions are granted to the NIC. Figure 14. EMC Unisphere Create NFS Export Wizard EMC VSPEX For Virtualized Oracle Database 11g OLTP 39
Chapter 4: Solution Implementation Configuring FAST Cache (optional) Enabling FAST Cache is a transparent operation to Oracle Database 11g R2 and no reconfiguration or downtime is necessary. To make the best use of either of the FAST technologies, EMC recommends that you first enable FAST Cache on the Oracle Database 11g R2 storage pool. For more details, refer to the EMC VSPEX for Virtualized Oracle Database 11g OLTP Design Guide. To create and configure FAST Cache, use the following steps: 1. Refer to the Reference Architecture paper entitled EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1,000 Virtual Machines for detailed steps about how to create FAST Cache. 2. In Unisphere, after creating the FAST Cache, click the Storage tab and select Storage Pool. Select Data Pool, and click Properties. 3. Select the Advanced tab in Storage Pool Properties and click Enabled to enable FAST Cache, as shown in Figure 15. 4. Click OK to complete the configuration. Note The FAST Cache feature on the VNX series array does not cause an instant performance improvement. The system must collect data about access patterns and promote frequently used information into the cache. This process can take a few hours during which the performance of the array steadily improves. Figure 15. Enabling FAST Cache in the Storage Pool Properties dialog 40 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Configuring FAST VP (optional) We created a three-tier FAST VP with a mixed storage pool, consisting of 5 flash drives, 25 SAS drives and 8 NL-SAS drives on VNX8000. FAST VP automatically relocates the LUN data from one tier to another within a pool. In this solution, we set the Auto-Tiering policy to Scheduled.For demonstration purpose, we configured the Data Relocation Schedule setting as Monday to Sunday, starting from 00:00 to 23:45, which determines the time window when FAST VP moves data between tiers. Note: The Data Relocation Rate and Data Relocation Schedule are highly dependent on the real workload in a customer environment. Usually, setting the Data Relocation Rate to Low has less impact on the current running workload. Set the Tiering Policy for all LUNs containing datafiles to Auto-Tier, so that FAST VP can automatically move the most active data to flash drive devices. Figure 16. FAST VP creating storage pool EMC VSPEX For Virtualized Oracle Database 11g OLTP 41
Chapter 4: Solution Implementation Initially, all datafiles were placed on SAS devices, as shown in Figure 17. Figure 17. Tier status before data movement With the workload running against the database for a few hours, FAST VP monitored and provided ongoing load balancing of LUNs across available drives until it reached a steady state, as shown in Figure 18. Figure 18. FAST VP in a steady state 42 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Example storage layout Figure 19 shows an example storage layout for VNX with FAST Cache or FAST VP enabled on the Oracle Database 11g R2 pool. Figure 19. Example virtualized oracle storage layout for VSPEX EMC VSPEX For Virtualized Oracle Database 11g OLTP 43
Chapter 4: Solution Implementation Implementing ESXi and vcenter This section documents the requirements for the installation and configuration of the ESXi hosts, vcenter configuration, and infrastructure servers required to support the architecture. Table 10 describes the tasks that must be completed. Table 10. Steps for installing and configuring ESXi and vcenter Task Description Reference documents Install ESXi Configure and deploy vcenter Server Configure ESXi networking Connect VMware datastores Create a virtual data center Apply vsphere license keys Install the ESXi 5.1 hypervisor on the physical servers that are being deployed for the solution. Install VMware vcenter Server 5.1 Configure ESXi networking including NIC trunking, VMkernel port, virtual machine port groups, and Jumbo Frames. Connect the VMware datastore to the ESXi hosts deployed for the solution. Create a virtual data center. Type the vsphere license keys in the vcenter licensing menu. vsphere Installation and Setup vcenter Server and Host Management Guide Installing vcenter Server 5.1 best practices vsphere Networking vsphere Storage vcenter Server and Host Management vsphere Installation and Setup Add ESXi hosts Connect vcenter to ESXi hosts. vcenter Server and Host Management Configure vsphere clustering Perform array ESXi host discovery Create a vsphere cluster Perform ESXi host discovery within the Unisphere console. vsphere Resource Management Using EMC VNX Storage with VMware vsphere TechBook Enable VMware High Availability (HA), DRS, and vmotion functionality Install EMC VNX USM Once you enable DRS, you may need to consider the use of VMware vsphere DRS Affinity and Anti-Affinity must rules for specific groups of virtual machines, for example, Oracle VMs may be restricted to a particular set of VMs due to licensing Install the EMC VNX USM on the administration console. Automating High Availability (HA) Services with VMware HA VMware Technical note EMC VSI for VMware vsphere: Unified Storage Management Product Guide Install the EMC VSI plug-in Install the EMC VSI plug-in on the administration console. EMC VSI for VMware vsphere: Unified Storage Management Product Guide For details of ESXi and vcenter, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1,000 Virtual Machines. 44 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Preparing the Oracle Database 11g R2 virtual machine template Overview VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required. Customization specifications, maintained in vcenter, further simplify the rollout of virtual machines. A deployment wizard, automation tool, or script can use these templates to automatically pre-create or amend server settings (such as server name, time zone, and network configuration) prior to building the new virtual machine. This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 11g R2 environments. To configure the template with the requirements and prerequisites for the Oracle software install, complete the following steps: 1. Create a virtual machine with the following resources: 2 vcpus 8 GB of vram 100 GB virtual disk 2. Install the operating system and RPM packages 3. Configure the system and kernel parameters 4. Create the OS groups and users 5. Install and detach the Oracle binaries ready for clone 6. Convert the virtual machine to a VMware template Creating the Oracle Database 11g R2 virtual machine Create a virtual machine on the ESXi server with the customer s guest OS configuration. Table 11 lists an example specification used when building the virtual machine template. Table 11. Virtual machine template specification Part CPU Memory Description 2 vcpus 8 GB Virtual disk 100GB 8GB swap partition 92GB bootable root partition Operating system Red Hat Enterprise Linux Server release 6.3 Kernel 2.6.32 EMC VSPEX For Virtualized Oracle Database 11g OLTP 45
Chapter 4: Solution Implementation Part Network interfaces OS users OS groups Oracle 11g R2 Binaries RPM packages installed System configuration Description Eth0: public/management IP network Eth1: Storage Network Username: oracle Group: oinstall, dba Oracle 11.2.0.3 installed and patched Refer to the Oracle Installation Guide for prerequisites See the Oracle Installation Guide for prerequisites. Completing the Oracle install prerequisites This section describes how to install and configure the operating system and Oracle prerequisites in the virtual machine prior to converting it to a template. Table 12. Oracle install prerequisites Task Description Reference Install the guest OS Install the Red Hat Enterprise Linux 6.3 64-bit operating system and configure the network interfaces. Red Hat Enterprise Linux 6 Installation Guide Install VMware tools Install the required RPM packages Set the kernel parameters Set the resource limits for the Oracle user Create the required software directories Configure HugePages Install VMware tools using Red Hat Package Manager (RPM) Install or update required RPM packages as listed in Table 13. Configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O. Check and adjust the resource limits for the Oracle software installation users Create the required directories for the Oracle software HugePages reduce the number of memory pages and reduces virtual memory management overhead. VMware knowledge Base article ID(1018392) Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations Table 13 lists the Red Hat Enterprise Linux 6 RPM packages required to install Oracle database 11g R2. 46 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Table 13. RHEL 6 RPM packages for Oracle Database 11g R2 binutils-2.20.51.0.2-5.11.el6 (x86_64) compat-libcap1-1.10-1 (x86_64) compat-libstdc++-33-3.2.3-69.el6 (x86_64) compat-libstdc++-33-3.2.3-69.el6.i686 gcc-4.4.4-13.el6 (x86_64) gcc-c++-4.4.4-13.el6 (x86_64) glibc-2.12-1.7.el6 (i686) glibc-2.12-1.7.el6 (x86_64) glibc-devel-2.12-1.7.el6 (x86_64)10 glibc-devel-2.12-1.7.el6.i686 ksh libgcc-4.4.4-13.el6 (i686) libgcc-4.4.4-13.el6 (x86_64) libstdc++-4.4.4-13.el6 (x86_64) libstdc++-4.4.4-13.el6.i686 libstdc++-devel-4.4.4-13.el6 (x86_64) libstdc++-devel-4.4.4-13.el6.i686 libaio-0.3.107-10.el6 (x86_64) libaio-0.3.107-10.el6.i686 libaio-devel-0.3.107-10.el6 (x86_64) libaio-devel-0.3.107-10.el6.i686 make-3.81-19.el6 sysstat-9.0.4-11.el6 (x86_64) On Oracle Linux 6, and Red Hat Enterprise Linux 6: unixodbc-2.2.14-11.el6 (x86_64) or later unixodbc-2.2.14-11.el6.i686 or later unixodbc-devel-2.2.14-11.el6 (x86_64) or later unixodbc-devel-2.2.14-11.el6.i686 or later EMC VSPEX For Virtualized Oracle Database 11g OLTP 47
Chapter 4: Solution Implementation Installing the Oracle Database 11g R2 binaries and patches As advised in the Design Guide, decide whether to install Oracle SE or Oracle EE Database 11g R2 software. Both editions use the same installation files. Table 14 includes the steps required to install the Oracle Database binaries and patches. Table 14. Installing the Oracle Database binaries and patches Task Description Reference Install the Oracle binaries Install the latest PSU and CPU Enable DNFS Detach the Oracle Home directory? Create the ORACLE_HOME cloning scripts Run the Oracle Universal installer and choose to Install choose database software only as shown in Figure 20. Install the latest database Patch Set Update and instructions available from the My Oracle Support website. A login ID and password as well as a support contract are required. Oracle Database is not shipped with Direct NFS enabled by default. Enable Direct NFS as shown in Figure 21. To remove the Oracle Database 11g R2 software from the Oracle Universal Installer inventory run the script $ORACLE_HOME/oui/bin/detachhome.sh To automate the cloning Oracle software process, create the scripts /root/root_clone.sh shown in Figure 23 Oracle Database Installation Guide 11g Release 2 (11.2) for Linux See Critical Patch Updates, Security Alerts and Third Party Bulletin Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 2 Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 6 As well as the official installation guide, Oracle Database Installation Guide 11g Release 2 (11.2) for Linux, Red Hat has produced a helpful whitepaper titled Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations which offers guidance specific to the Red Hat operating system. Figure 20. Oracle Universal Installer Install choose database software only 48 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, complete the following steps: Change the directory to $ORACLE_HOME/rdbms/lib. cp libodm11.so libodm11.so_stub ln -s libnfsodm11.so libodm11.so Enter the following command: make -f ins_rdbms.mk dnfs_on Figure 21. Enable Oracle Direct NFS client [root@vspex-ora02 ~]# cat /root/root_clone.sh #!/bin/bash OHOME=/u01/app/oracle/11.2.0.3 echo "Cloning Oracle Binaries as Oracle at $(date +%Y%m%d%H%M)" su - oracle -c "$OHOME/oui/bin/cloneHome.sh" echo "Clone Finished at $(date +%Y%m%d%H%M)" echo "Running root.sh at $(date +%Y%m%d%H%M)" $OHOME/root.sh echo "Install Finished at $(date +%Y%m%d%H%M)" Figure 22. Example of script root_clone.sh [root@vspex-ora02 ~]# cat $OHOME/oui/bin/cloneHome.sh #!/bin/sh OHOME=/u01/app/oracle/11.2.0.3 OHOMENAME=OraDb11g_home1 OBASE=/u01/app/oracle CUR_DIR=`pwd` cd $OHOME/oui/bin ${OHOME}/clone/bin/clone.pl ORACLE_HOME="${OHOME}" ORACLE_HOME_NAME="${OHOMENAME}" ORACLE_BASE="${OBASE}" -ignoresysprereqs cd $CUR_DIR Figure 23. Example of script clonehome.sh EMC VSPEX For Virtualized Oracle Database 11g OLTP 49
Chapter 4: Solution Implementation Preparing the Oracle Database 11g R2 virtual machine template Convert the virtual machine to a template called vspx-orac11203ee-2vcpu-8gb-tmplt as shown in Error! Reference source not found.. Figure 24. Virtual machine template example Customizing the VMware template Create a custom specification, Oracle_2vCPU_8GB_11203_EE_spec, using the vcenter client. This requests the new virtual machine name and sets place holder IP addresses for the network interfaces as shown in Figure 25. Figure 25. vsphere custom specification example 50 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Deploying the Oracle Database 11g R2 virtual machine Overview Table 15 shows the mapping of the compute requirements for the Oracle Database 11g R2 virtual machines. Table 15. Oracle Database 11g R2 reference virtual machine model Virtual Machine Compute Requirement for Oracle Database 11g R2 v C Memory in GB P U Total reference virtual machines Small 2 8 4 Medium 4 16 8 Large 8 32 16 Deploying new virtual machines There are serveral methods available to deploy a virtual machine from a template using custom specifications: By manually choosing Deploy Virtual Machine from this Template in vcenter Choosing Customize using an existing customization specification By creating a workflow with an orchestration tool such as vcenter Orchestrator By creating a custom PowerCLI script that can be edited or parameterized The example PowerCLI script in Figure 26: Assigns variables for: template name, custom specification name, virtual machine name, ESXi host name, VMFS datastore name, virtual machine public IP address, virtual machine storage IP address, and the number of vcpus, and vram size in MB. Assigns the virtual machine public and storage IP addresses to the two vnics included in the custom specification in Figure 26. Creates the new virtual machine using the template and custom specification. Changes the CPU and VRAM assignment for the virtual machine. Starts the virtual machine. EMC VSPEX For Virtualized Oracle Database 11g OLTP 51
Chapter 4: Solution Implementation # Connect to Vcenter Connect-VIServer -server 192.168.28.33 -user Administrator -password <Password> { #Assign Variables for template, specification, VM, ESXi Host, Datastore # IP addresses for the Public and Storage network interfaces # and the vcpu and vram assigned to the Virtual Machine $Template="vspx-Orac11203EE-2vcpu-8GB-tmplt" $Customization="Oracle_2vCPU_8GB_11203_EE_spec" $VMName="<Virtual Machine Name>" $VMHost=" <ESXi Host Name>" $Datastore="<Datastore Name>" $IPAdressPublic= < Public IP Address> $IPAdressStorage= <Storage IP Address> $NumVCPU= 4 $vrammb= 16384 # Assign IP addresses to the two vnics included in the custom specification Get-OSCustomizationSpec $Customization Get-OSCustomizationNicMapping where { $_.Position -eq '1'} Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressPublic -Subnetmask 255.255.255.0 -DefaultGateway 192.168.28.1 Get-OSCustomizationSpec $Customization Get-OSCustomizationNicMapping where { $_.Position -eq '2'} Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressStorage -Subnetmask 255.255.255.0 -DefaultGateway 192.168.40.1 # Create the new VM using the Template and custom spec New-VM -Name $VMName -OSCustomizationSpec $Customization -Template $Template - VMHost $VMHost -Datastore $Datastore # Change the CPU and memory assignment for the VM Set-VM $VMName -NumCPU $NumVCPU -MemoryMB $vrammb -Confirm:$false # Start up the VM Start-VM -VM $VMName -Confirm:$false } Figure 26. Example PowerCLI script to deploy a virtual machine from template Edit and run the example PowerCLI script, or edit it to read a parameter file and deploy multiple virtual machines. Cloning the Oracle binaries ona new virtual machine Run the clone process and reattach the Oracle home directory to edit and run a second PowerCLI to complete the virtual machine deployment. This script, shown in Figure 27, calls the root_clone.sh from Figure 22 which does this. 52 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation { $script="/root/root_clone.sh" $huser="root" $hpwd="<root Password>" $VMName="<Virtual Machine Name>" Invoke-VMScript -ScriptText $script -VM $VMName -GuestUser $huser -GuestPassword $hpwd -ScriptType bash } Figure 27. Example PowerCLI script to clone and reattach the ORACLE_HOME Determining the database prerequisites Use the VSPEX for Virtualized Oracle Qualification Worksheet in the Design Guide to size the virtualization infrastructure and the overall NFS filesystems in which to store the Oracle database. These documents also provide guidance on the Oracle initialization parameters required by the database. Table 15 on page 51 shows the sizing of the virtual machine. Determine the size of the individual datafiles based on the schema design and tablespace placement of tables and indexes. Datafile placement is covered in the section Configuring the database file layout for NFS. For guidance on configuring the initialization parameters, see the Design Guide and the Configuring the initialization parameters section of this document. Before creating the database, you must create a listener process to receive the incoming client connection requests using the Oracle Net Configuration Assistant (NETCA) utility, as shown in Figure 28. Figure 28. Oracle Net Configuration Assistant utility EMC VSPEX For Virtualized Oracle Database 11g OLTP 53
Chapter 4: Solution Implementation Consider these items before creating an Oracle 11g R2 database: Select the required database character set to meet your localization and globalization needs. Consider the time zones your database must support. Ensure the SYSTEM and SYSAUX tablespace are created as Locally Managed Tablespaces, appropriately sized, and have AUTOEXTEND enabled. Plan to use an undo tablespace to manage your undo data. Configuring the database file layout for NFS In this example, the components of the Oracle Database: datafiles, online redo log files, temp files, control files, and FRA files reside on NFS file systems. These file systems are designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file. Table 16 lists each file type and indicates where they reside. Table 16. Database file layout for NFS Content Datafiles Temp files Online redo logs, control files FRA files Location /u02/app/oracle/oradata/${oracle_sid} /u04/app/oracle/oradata/${oracle_sid} /u03/app/oracle/oradata/${oracle_sid} /u05/app/oracle/oradata/${oracle_sid} This solution uses the Oracle Direct NFS client, which offers improved performance over the standard NFS client supplied with the operating system. By default, Direct NFS serves the mount entries found in the file /etc/mtab, which lists all currently mounted filesystems along with their initialization options. This should match the system configuration file /etc/fstab where the mount options are defined. Figure 29 provides a sample /etc/mtab with NFS mount options: NFSServer1:/vspex_ora2_data /u02/app/oracle/oradata/vspex1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 NFSServer2:/vspex_ora2_redo /u03/app/oracle/oradata/vspex1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.10 0 0 NFSServer1:/vspex_ora2_temp /u04/app/oracle/oradata/vspex1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 NFSServer1:/vspex_ora2_fra /u05/app/oracle/oradata/vspex1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 Figure 29. Example /etc/mtab NFS mount options Oracle recommends that you set the NFS buffer size parameters rsize and wsize to 32768 in the system configuration file /etc/fstab. However, you can use the file oranfstab to specify additional Oracle Database specific options for Direct NFS. 54 EMC VSPEX For Virtualized Oracle Database 11g OLTP
The oranfstab is located in one of the two following locations: Chapter 4: Solution Implementation /etc/oranfstab where the options are global to all Oracle databases on the server $ORACLE_HOME/dbs/oranfstab where the options are specific to a single database The Oracle Database requires that mount points be mounted by the kernel NFS system even when served through Direct NFS. To enable Direct NFS, you can optionally create an oranfstab file with the following attributes for each NFS server to be accessed using Direct NFS: Server: The NFS server name. Path: Up to four network paths to the NFS server, specified either by IP address, or by name, as displayed using the ifconfig command on the filer. Local: Up to four local paths on the database host, specified by IP address or by name, as displayed using the ifconfig command run on the database host. Export: The exported path from the NFS server. Mount: The corresponding local mount point for the exported volume. Dontroute: Specifies that outgoing messages should not be routed by the operating system, but instead sent using the IP address they are bound to. Please note that this attribute does not work on Linux with multiple paths in the same subnet. mnt_timeout: Specifies (in seconds) how long Direct NFS client should wait for a successful mount before timing out. This parameter is optional and the default timeout is 10 minutes. Figure 30 provides an example of an oranfstab file with two NFS server entries: server: NFSServer1 local: vspex_ora2 path: NFSServer1 export: /vspex_ora2_data mount: /u02/app/oracle/oradata/vspex1 export: /vspex_ora2_temp mount: /u04/app/oracle/oradata/vspex1 export: /vspex_ora2_fra mount: /u05/app/oracle/oradata/vspex1 server: NFSServer2 local: vspex_ora2 path: NFSServer2 export: /vspex_ora2_redo mount: /u03/app/oracle/oradata/vspex1 Figure 30. Oranfstab example EMC VSPEX For Virtualized Oracle Database 11g OLTP 55
Chapter 4: Solution Implementation Configuring the database file layout for Block Red Hat 6.3 supports multiple and various file systems, like VFAT, ext2,ext3, ext4 and Reiser file system. Oracle generally does not certify file systems, but Linux is a specific case.the current support includes ext2, ext3, ext4 (Oracle Linux 5.6 and later),etc. The ext4 is the default file system in this solution. Logical Volume Management (LVM) can be used to allocate disk space and provide disk striping. Formatting Physical Volumes (PVs) To initialize a disk or disk partition as a physical volume we just run the "pvcreate" command on the whole disk or the partition. For example: pvcreate /dev/sdb or pvcreate /dev/sdb1 This creates a volume group descriptor at the start of the second SCSI disk or the /dev/sdb1 partition. Creating Volume Groups (VGs) Use the "vgcreate" command to group selected PVs into VGs. The following command creates a volume group named " data_volume_group" from five disk partitions from different disks: vgcreate data_volume_group /dev/sdb1 /dev/sdc1 /dev/sdf1 /dev/sdd1 /dev/sde1 To activate a single volume group data_volume_group, use the command: vgchange -a y /dev/data_volume_group Creating Logical Volumes (LVs) The following is an example of creating logical volume data_vg-data_lv from the volume group data_volume_group. This command creates a 2048G LV named datavg-data_lv for the vg named data_volume_group and its block device special file /dev/ data_volume_group/ data_vg-data_lv lvcreate -i5 -I4 -L2048G -n data_vg-data_lv data_volume_group Where size is either "-l num_extents" or "-L num_bytes", where num_bytesis a number followed by one of k, m, g, or t. One of the most common options is "-n name" (you can use "--name" for "-n") to specify a name for the logical volume. Another option includes "-i num_stripes -I stripe_size" to create an LV with striped mapping (stripe_size is a number between 2 and 9) Once the LVs have been created you can format them with filesystems (or as swap space) using standard tools such as "mkfs". One example of formatting LV with ext4 filesystem: mkfs.ext4 -E stride=16 /dev/data_volume_group/data_vg-data_lv 56 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation If the new filesystem can be successfully mounted, a final step is to edit the /etc/fstab file and possibly the rc.sysinit file, so that the volumes are mounted automatically at boot time. Figure 31 shows an example of /etc/fstab. Figure 31. Example of /etc/fstab For more information, see My Oracle Support Note ID 1476869.1 (Available Mount Options to Improve ext4 Filesystem Performance) and 236826.1 (Supported and Recommended File Systems on Linux). Configuring the initialization parameters To configure the initialization parameters, you can use one of the following methods. Automatic Shared Memory Management To automate the management of shared memory structures, implement Automatic Shared Memory Management (ASMM). The following shared memory structures will be automatically managed: DB_CACHE_SIZE SHARED_POOL_SIZE LARGE_POOL_SIZE JAVA_POOL_SIZE STREAMS_POOL_SIZE When implemented, ASMM enables the Oracle database to distribute memory between these components based on workload requirements. Hence, these components are considered autotuned parameters. The background process, Memory Manager (MMAN), coordinates the sizing of these components and moves memory to where it is needed most. To implement this feature, set the following initialization parameters: SGA_TARGET set to a nonzero value STATISTICS_LEVEL=TYPICAL (or ALL) Automatic Memory Management Do not use Oracle Automatic Memory Management (AMM) since AMM is incompatible with HugePages. If you want to use HugePages make sure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set. For more information, see My Oracle Support Note ID 749851.1. EMC VSPEX For Virtualized Oracle Database 11g OLTP 57
Chapter 4: Solution Implementation I/O operations for file system files To enable direct I/O and async I/O for file system files, set FILESYSTEMIO_OPTIONS=SETALL With async I/O, processing continues while the I/O request is submitted and processed. This allows async I/O to bypass some of the performance bottlenecks associated with I/O operations. Direct I/O is a feature available in the filesystem that delivers data directly to the application without caching in the file system buffer cache. Direct NFS does not depend on the value of FILESYSTEMIO_OPTIONS. Direct NFS always issues async and direct I/O as it does not depend on OS support. However, so that there is a backup to the OS NFS client in case of mis-configuration, set filesystemio_options to SETALL if the OS supports it. Enabling the HugePages setting HugePages was enabled in the Linux Kernel as part of the Template creation process described in Table 12 on page 46. The database initialization parameter, USE_LARGE_PAGES, was involved from 11.2.0.2 to manage HugePages for use by the database. From version 11.2.0.3, setting USE_LARGE_PAGES to AUTO causes the oradism process to try to reconfigure the Linux kernel to increase the number of HugePages to match the database requirements. See My Oracle Support article USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] for more details. Creating the database using DBCA The Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready to use when DBCA completes. Launch DBCA as a standalone tool to create an Oracle Database. To start DBCA on UNIX, Linux, or at a Windows command-line prompt, enter the following command: dbca Note: The dbca utility is typically located in the ORACLE_HOME/bin directory. 58 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation Figure 32. Oracle Database Configuration Assistant setup screen Oracle DBCA runs in interactive mode. Using the screens in Oracle DBCA, you can either modify one of the preconfigured database types or customize the database. General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation. This involves amending the Initialization parameters Size and placement of the database files to use the NFS mount points Performing the database post creation activities After you create a database as described in Creating the database using DBCA, the instance is open and available for normal database use. However, the Oracle database will not start up or shut down automatically with the virtual machine. With Oracle Database 11g R2, there are two methods to automate the Oracle Database startup and shutdown process: Use an initialization script and the /etc/oratab file as defined by My Oracle Support Document How to Automate Startup/Shutdown of Oracle Database on Linux [ID 222813.1] Install the additional 11.2 Grid Infrastructure software using the single instance option. This will use a modified Cluster Ready Services (CRS) component to automate the startup of the database, listener, and Oracle Automatic Storage Management (a grid Infrastructure component) if required. This solution uses the initialization script shown in Figure 33. Follow these steps to set up automated startup and shutdown of the database and listener. 1. Update the /etc/oratab file with an entry in the following format: <ORACLE_SID>:<ORACLE_HOME>:Y Where: ORACLE_SID is the name of the database EMC VSPEX For Virtualized Oracle Database 11g OLTP 59
Chapter 4: Solution Implementation ORACLE_HOME is the directory where the Oracle 11g R2 database binaries are installed Y states that the database can be started up and shutdown automatically For example: VPLEXM1: /u01/app/oracle/11.2.0.3:y 2. Figure 33 shows an example of how to create the service script /etc/init.d/dbora as root user. #!/bin/bash # # chkconfig run levels and script numbering # chkconfig: 35 99 10 # description: Starts and stops Oracle processes # # Set ORA_HOME to be equivalent to the $ORACLE_HOME # from which you wish to execute dbstart and dbshut; # Set ORA_OWNER to the user id of the owner of the # Oracle database in ORA_HOME. # ORA_HOME=/u01/app/oracle/11.2.0.3 ORA_OWNER=oracle case "$1" in 'start') # Start the TNS Listener su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start" # Start the Oracle databases: su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart"& touch /var/lock/subsys/dbora ;; 'stop') # Stop the TNS Listener su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl stop" # Stop the Oracle databases: su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut" rm -f /var/lock/subsys/dbora ;; esac # End of script dbora Figure 33. Oracle startup/shutdown script /etc/init.d/dbora 3. Execute the script: chmod 755 /etc/init.d/dbora 60 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 4: Solution Implementation 4. Register the service to the Linux service mechanism for automatic startup at run level 3 and 5 by issuing the following command. /sbin/chkconfig --add dbora With the infrastructure build tasks complete, the database is now ready for use. Implementing backup and recovery Overview For information about implementing backup and recovery for an Oracle 11gR2 Database, refer to the document entitled EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and s available on EMC s online support. EMC VSPEX For Virtualized Oracle Database 11g OLTP 61
Chapter 5: Solution Verification Chapter 5 Solution Verification This chapter presents the following topics: Verifying the baseline infrastructure... 64 Verifying the Oracle Database... 66 Backup and recovery verification methodology... 68 EMC VSPEX For Virtualized Oracle Database 11g OLTP 63
Chapter 5: Solution Verification Verifying the baseline infrastructure Overview This section provides a list of items that you need to review after configuring the solution. The goal of this section is to verify the configuration and functionality of the solution, and to ensure that the configuration supports the core availability requirements. Table 17 describes the tasks that must be completed. Table 17. Tasks for verifying the VSPEX Proven Infrastructure installation Task Description Reference Verify ESXi functionality Verify the basic ESXi functionality of the solution with a post-installation checklist. Using EMC VNX storage with VMware vsphere Verify redundancy of the solution components Verify the Oracle Database configuration in the VSPEX Proven Infrastructure Monitor the health of the VSPEX Proven Infrastructure for virtualized Oracle Database Verify the redundancy of the solution of the components: Storage ESXi host Network switch Verify the Oracle Database configuration. Use the tool to monitor the VSPEX Proven Infrastructure s health http://www.vmware.com Vendor documentation http://support.oracle.com View alert logs (Oracle Database 11g R2) View storage status from Unisphere management console View VM logs from vsphere 5.1 client Verifying the ESXi functionality EMC recommends that you verify the ESXi configurations before deploying the system to production on each ESXi server. For more detailed information, refer to appropriate documents accessible from the VSPEX Proven Infrastructure Launch Page. Verifying the solution component redundancy To ensure that the various components of the solution maintain availability requirements, it is important to test specific scenarios related to maintenance or hardware failure. EMC recommends that you verify redundancy of the solution components including storage, ESXi hosts, and network switches. For detailed steps, refer to the appropriate documents accessible from the VSPEX Proven Infrastructure Launch Page. 64 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 5: Solution Verification Verifying the Oracle Database configuration Using the health monitoring tools To ensure that Oracle Database 11g R2 is running smoothly in the VSPEX infrastructure, use these steps to verify and monitor the Oracle Database configuration: 1. Check the installed Oracle Database contents and directory location. Make sure Oracle Database has been installed successfully. 2. Verify that the Oracle Database that you plan to use is accessible. Use the TNSPING utility to test connectivity from the client with the following command: tnsping net_service_name count. Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for virtualized Oracle Database. Table 18 lists some tools for you to monitor and troubleshoot your VSPEX solution. Table 18. Tools esxtop Tools for monitoring the solution Description The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi Hypervisor performance metrics, such as CPU. It is important that you understand the performance for hypervisor in order measure or troubleshoot the Oracle Database performance-related issues. Unisphere Analyzer Oracle Enterprise Manager Database Control Automatic Workload Repository (AWR) Unisphere Analyzer software is preloaded on all VNX storage systems with the FLARE/block bundles (or later). It is an easy to monitor, collect, and analyze the all aspects of storage related data from it. Oracle Enterprise Manager Database Control (Database Control) is the primary tool for managing your Oracle database. It is installed with the database. Using Database Control, you can perform administrative tasks such as creating schema objects (tables, views, indexes, and so on), managing user security, managing database memory and storage, backing up and recovering your database, and importing and exporting data. You can also view performance and status information about your database. The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and selftuning purposes. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views. EMC recommends that you use the above tools to monitor the performance of the entire Oracle Database environment. For details about performance verification and test methodology, refer to the companion Design Guide. EMC VSPEX For Virtualized Oracle Database 11g OLTP 65
Chapter 5: Solution Verification Verifying the Oracle Database Overview This section provides a verification example of an Oracle OLTP Database in the VSPEX Proven Infrastructure for virtualized Oracle Database solution. The purpose is to measure the performance of the Oracle Database to ensure that the VSPEX Proven Infrastructure meets the customer s business requirements. Before you verify your own VSPEX Proven Infrastructure, EMC recommends that you read the Application validation methodology section in the companion Design Guide to understand the test methodology. Understanding the key metrics After you identify the business requirements, you need to know the key metrics for the Oracle Database testing when running the Oracle Database validation tests. The key metrics are identified in the Design Guide in the Application validation methodology section. My Oracle Support Note 1275596.1 provides a list of typical acceptable I/O response times. Loading the test configuration The Swingbench Order Entry PL/SQL (SOE) schema delivers the OLTP workloads required by the solution. The Order Entry schema introduces heavy contention on a small number of tables and is designed to stress CPU and memory. To establish an optimal configuration, we tested different users accessing the order entry schema. Before running an Oracle Database 11g R2 test, it is important to understand the test scenarios according to the business requirements. You can easily create your own test scenarios using Swingbench. By editing the transaction list, you can define the different operations that end users perform in the Oracle Database. Table 19 lists some of the general operations we considered in the Oracle Database 11g R2 VSPEX solution. Table 19. General operations Workload Operation Weight Swingbench Order Entry Customer Registration 30 Browse Products 60 Order Products 50 Process Orders 30 Browse Orders 10 Swingbench allows you to select a variable think time (delay) between transactions. Figure 34 shows the load panel on the Swingbench interface, with the appropriate think time parameters set. 66 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 5: Solution Verification Figure 34. Swingbench interface load panel screenshot We simulated the workload using Swingbench against the database as shown in Figure 35. Figure 35. Swingbench order entry screen for workload simulation Once the test is complete, you can use the Automatic Workload Repository (AWR) report to check the results and verify that you have achieved the key metrics in your test environment. Since the metrics can vary with different kinds of Oracle Database configuration and VSPEX offerings for VMware, refer to the VSPEX Sizing Tool for the detailed metrics. If the test results are not ideal, it is easy to identify the bottleneck from the AWR report. For detailed information about the AWR report, refer to the My Oracle Support Note ID 1359094.1. After identifying potential bottlenecks, refer to the Implementing the storage section to make sure that you have the correct configuration of VNX, switches, hypervisor, or load balancer for the VSPEX for virtualized Oracle solution. System bottlenecks represent a point of contention where there are insufficient resources to service user transaction requests. Consider adding more resources into the solution to meet the requirement. Often, the reason for the bottleneck is inefficient custom code or thirdparty solutions due to customization. EMC VSPEX For Virtualized Oracle Database 11g OLTP 67
Chapter 5: Solution Verification Reviewing the Oracle dataset In this VSPEX validation test, we used the Swingbench Order Entry - PL/SQL (SOE) schema to deliver the OLTP workloads. This schema models a traditional OLTP database. Tables and indexes reside in separate tablespaces. The SOE schema used in this solution contains the tables and indexes shown in Table 20. Table 20. Table name CUSTOMERS Schema tables and indexes Index CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_EMAIL_IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX INVENTORIES ORDERS ORDER_ITEMS PRODUCT_DESCRIPTIONS PRODUCT_INFORMATION WAREHOUSES INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX PRD_DESC_PK (UNIQUE), PROD_NAME_IX PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX WAREHOUSES_PK (UNIQUE) LOGON Backup and recovery verification methodology To verify your backup and recovery methodology, refer to the EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide s. 68 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 6: Reference Documentation Chapter 6 Reference Documentation This chapter presents the following topics: EMC documentation... 70 Other documentation... 71 Links... 71 EMC VSPEX For Virtualized Oracle Database 11g OLTP 69
Chapter 6: Reference Documentation EMC documentation EMC EMC VSPEX Private Cloud: VMware vsphere 5.1 for up to 1000 Virtual Machines Reference Architecture White Paper: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and VNX System Installation Guide VNX Series Configuration Worksheet Using EMC VNX Storage with VMware vsphere TechBook EMC VSI for VMware vsphere: Unified Storage Management Product Guide EMC VSI for VMware vsphere: Unified Storage Management Product Guide VNX family http://www.emc.com/storage/vnx/vnx-family.htm VNX series documentation on EMC online support site https://support.emc.com/products/12781_vnx-series/documentation/ Oracle Oracle Database 11g R2 Documentation Library 11g Release 2 (11.2) http://www.oracle.com/pls/db112/homepage http://support.oracle.com Oracle Database 11g Release 2 documentation: http://docs.oracle.com/cd/e11882_01/install.112/e24321.pdf http://www.oracle.com/technetwork/indexes/documentation/index.html Oracle Database Release Notes for Linux My Oracle Support https://support.oracle.com/ latest OPatch and latest PSU HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux [Article ID 749851.1] USE_LARGE_PAGES To Enable HugePages [Article ID 1392497.1] HugePages on Oracle Linux 64-bit [Article ID 361468.1] FAQ: How to Use AWR Reports to Diagnose Database Performance Issues [Article ID 1359094.1] VMware vsphere Installation and Setup vcenter Server and Host Management Guide Installing vcenter Server 5.1 best practices vsphere Networking vsphere Storage vcenter Server and Host Management 70 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Chapter 6: Reference Documentation vsphere Installation and Setup vcenter Server and Host Management vsphere Resource Management Automating High Availability (HA) Services with VMware HA VMware Technical note General documentation: http://www.vmware.com VMware vsphere Documentation: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html vsphere PowerCLI Documentation: http://www.vmware.com/support/developer/powercli/ Other documentation Links Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations http://www.redhat.com/resourcelibrary/reference-architectures/deployingoracle-11g R2-on-rhel-6 Note: The links provided were working correctly at the time of publication. Everything Oracle at EMC https://community.emc.com/community/connect/everything_oracle My Oracle Support: Oracle Support Website https://support.oracle.com (user account required) FAQ: How to Use AWR reports to Diagnose Database Performance Issues [ID 1359094.1] Requirements for Installing Oracle 11g R2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1] How to Tell if the IO of the Database is Slow [Article ID 1275596.1] HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1] USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] HugePages on Oracle Linux 64-bit [ID 361468.1] Installing and Using Standby Statspack in 11g [ID 454848.1] Installing and Configuring StatsPack Package[Article ID 149113.1] EMC VSPEX For Virtualized Oracle Database 11g OLTP 71
Appendix A: Configuration Worksheet Appendix A Configuration Worksheet This appendix presents the following topic: Pre-configuration worksheet for VSPEX... 73 72 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Appendix A: Configuration Worksheet Pre-configuration worksheet for VSPEX Before configuring Oracle Database 11g R2 for this solution, you need to gather some customer-specific configuration information such as IP addresses, hostnames, and so on. The following tables provide a worksheet that you can use to record the information. You can also print and use the worksheet as a customer leave behind document for future reference. To confirm the customer information, cross-reference with the relevant array configuration worksheet: EMC VNX File and Unified Worksheets. Table 21. Common server information Server Name Purpose Primary IP Domain Controller DNS Primary DNS Secondary DHCP NTP SMTP SNMP vcenter Server Swingbench server Database server 01 Database server 02 Database server 03 Table 22. ESXi server information Server Name Purpose Primary IP Private Net (storage) addresses ESXi Host 1 ESXi Host 2 EMC VSPEX For Virtualized Oracle Database 11g OLTP 73
Appendix A: Configuration Worksheet Table 23. Array information Array name Root password NAS admin password CS0 Primary hostname CS1 Secondary hostname (if applicable) CS0 Primary IP address CS1 Secondary IP address (if applicable) SPA IP address SPB IP address Virtual machine infrastructure pool name Virtual machine infrastructure file system name Datafile storage pool name Datafile file system name Redo log storage pool name Redo log file system name Fra storage pool name Fra file system name Table 24. Network infrastructure information Name Purpose IP Subnet Mask Ethernet Switch 1 Default Gateway Ethernet Switch 2 Table 25. VLAN information Name Network Purpose VLAN ID Allowed Subnets Virtual Machine Networking Management NFS networking vmotion 74 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Appendix A: Configuration Worksheet Table 26. Account Service accounts Purpose vcenter Server administrator Array administrator Oracle Database administrator Password (optional, secure appropriately) EMC VSPEX For Virtualized Oracle Database 11g OLTP 75
Appendix A: Configuration Worksheet 76 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Appendix B: Example Virtual Disk Layout Appendix B: Example Virtual Disk Layout This chapter presents the following topic: Appendix B: Example Virtual Disk Layout 77 Example Virtual Disk Layout of Oracle Database on the VMFS Datastore... 78 EMC VSPEX For Virtualized Oracle Database 11g OLTP 77
Appendix B: Example Virtual Disk Layout Example Virtual Disk Layout of Oracle Database on the VMFS Datastore For customers who plan to virtualize Oracle database on the VMware platform, storage layout guidelines for Oracle database on VMware technology, with respect to the number of LUNs and maximizing I/O performance, are the same as with a physical installation. Table 27 shows an example of virtual disk layout. Table 27. Example Virtual Disk Layout Datastore Name OSDATASTORE VMFSDATA01 VMFSDATA02 VMFSDATA03 VMFSDATA04 VMFSDATA05 VMFSREDO01 VMFSREDO02 VMFSREDO03 VMFSREDO04 VMFSREDO05 VMFSFRA01 Virtual Disk on ESX VMDK- hard disk 1 VMDK- hard disk 2 VMDK- hard disk 3 VMDK- hard disk 4 VMDK- hard disk 5 VMDK- hard disk 6 VMDK- hard disk 7 VMDK- hard disk 8 VMDK- hard disk 9 VMDK- hard disk 10 VMDK- hard disk 11 VMDK- hard disk 12 Guest OS Device Name Virtual device Virtual SCSI Driver VMDK Size (GB) Purpose /dev/sda SCSI 0:0 Paravirtual 100 Redhat Linux 6.3 OS and Oracle binaries /dev/sdb1 SCSI 1:0 Paravirtual 500 Database datafile /dev/sdc1 SCSI 1:1 Paravirtual 500 Database datafile /dev/sdd1 SCSI 1:2 Paravirtual 500 Database datafile /dev/sde1 SCSI 1:3 Paravirtual 500 Database datafile /dev/sdf1 SCSI 1:4 Paravirtual 500 Database datafile /dev/sdg1 SCSI 2:0 Paravirtual 100 Database redo file /dev/sdh1 SCSI 2:1 Paravirtual 100 Database redo file /dev/sdi1 SCSI 2:2 Paravirtual 100 Database redo file /dev/sdj1 SCSI 2:3 Paravirtual 100 Database redo file /dev/sdk1 SCSI 2:4 Paravirtual 100 Database redo file /dev/sdl1 SCSI 3:0 Paravirtual 500 Database FRA file 78 EMC VSPEX For Virtualized Oracle Database 11g OLTP
Appendix B: Example Virtual Disk Layout Datastore Name VMFSFRA02 VMFSFRA03 VMFSFRA04 VMFSFRA05 Virtual Disk on ESX VMDK- hard disk 13 VMDK- hard disk 14 VMDK- hard disk 15 VMDK- hard disk 16 Guest OS Device Name Virtual device Virtual SCSI Driver VMDK Size (GB) Purpose /dev/sdm1 SCSI 3:1 Paravirtual 500 Database FRA file /dev/sdn1 SCSI 3:2 Paravirtual 500 Database FRA file /dev/sdo1 SCSI 3:3 Paravirtual 500 Database FRA file /dev/sdp1 SCSI 3:4 Paravirtual 500 Database FRA file EMC VSPEX For Virtualized Oracle Database 11g OLTP 79