Planning and Configuring HUAWEI OceanStor V3 Converged Storage Systems to Maximize OLAP Oracle Database's Performance and Availability

Transcription

1 Planning and Configuring HUAWEI OceanStor V3 Converged Storage Systems to Maximize OLAP Oracle Database's Performance and Availability This document is aimed at the scenario where HUAWEI OceanStor V3 converged storage systems are used to serve Online Analytical Processing (OLAP) Oracle Database 12c. This document focuses on how to efficiently deploy Oracle Database based on OceanStor V3 converged storage systems, and verifies OLAP applications in typical enterprise database scenarios. The best practices described in this document help you obtain higher application deployment efficiency and better application operation quality, thereby ensuring Oracle Database's performance and availability. Wang Yaohui Storage Solutions, IT, Huawei Enterprise BG V1.0 Huawei Technologies Co., Ltd.

2 Contents 1 Overview Introduction Purpose Intended Audience Business Scenario Workload Model Acronyms and Abbreviations Products and Technologies OceanStor V3 Converged Storage Systems Next-Generation Hardware Multi-Controller Architecture Convergence Design Smart Software Unified and Easy Management RAID 2.0+ Block Virtualization Oracle Database and Cluster Oracle RAC and ASM Oracle System Architecture Oracle Application Types Best Practices for Oracle Database Planning and Configuration SAN Networking Setting Zones and vlans Storage Configuration Planning Disk Domains Storage Pools LUNs Mapping Views Host Configuration Queue Depth I/O Alignment Block Device Scheduling Algorithms Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 2

3 3.3.4 Block Device I/O Settings Database Configuration Database Parameters Online Logs Backup and Archiving Example of Oracle Database Planning and Configuration Solution Architecture Database Plan Storage Configuration Host Configuration Database Configuration Workload Verification Results Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 3

4 1 Overview 1.1 Introduction This document is aimed at the scenario where HUAWEI OceanStor V3 converged storage systems are used to serve Online Analytical Processing (OLAP) Oracle Database 12c. This document focuses on how to efficiently deploy Oracle Database based on OceanStor V3 converged storage systems, and verifies OLAP applications in typical enterprise database scenarios. The best practices described in this document help you obtain higher application deployment efficiency and better application operation quality, thereby ensuring Oracle Database's performance and availability. 1.2 Purpose This document describes the plan and configuration for the scenario where OceanStor V3 converged storage systems are used to serve OLAP Oracle Database. It provides reference for Huawei partners and customers, in order to help them simplify IT system planning and deployment and reduce O&M risks. 1.3 Intended Audience This document is intended for Huawei IT engineers and partners. It provides the best practices of Oracle OLAP for IT partners, storage architects, database architects, and IT system administrators who want to deploy Oracle OLAP applications based on HUAWEI OceanStor V3 converged storage systems. It is assumed that the readers are familiar with the following products and technologies: Storage systems Oracle Database Linux operating systems 1.4 Business Scenario Database services are typically classified into two types: OLTP and OLAP. OLTP is commonly seen in traditional relational databases. It mainly involves basic and daily transaction processing, such as stock exchanges and bank transactions. OLAP is commonly seen in data warehouse systems. It supports complex analytical operations and provides Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 4

5 graphical and easy-to-understand query results to help decision makers obtain accurate business operation information, so that they can make correct decisions. With the wide use of database technology, enterprise information systems generate a large amount of data. How to extract useful information from massive data to assist decision making has become a great challenge facing decision makers of enterprises. How to accelerate complex query and analysis in OLAP applications is the key to addressing the challenge. Oracle Database is one of the most commonly used databases and one of the most important application scenarios for storage devices. Deploying OLAP Oracle Database on OceanStor V3 storage systems can ensure service reliability and improve database analysis and query performance, helping enterprises make correct decisions. This document describes the recommended plan and configuration, including networking, storage plan and configuration, as well as host and database parameter settings, for the scenario where OLAP Oracle Database is deployed based on an OceanStor V3 converged storage system, in order to help customers mitigate performance and availability risks in IT systems. 1.5 Workload Model This document uses a TPCH-Like benchmark test model that consists of a suite of business-oriented queries and concurrent data modifications. The queries have been chosen to have broad industry-wide relevance. This benchmark illustrates the following aspects of decision support systems: Examining large volumes of data Executing queries with a high degree of complexity Giving answers to critical business questions The test model defines eight tables, recording information about projects, orders, suppliers, and customers. The workload test procedure includes loading, analysis, and updating operations. In the loading phase, SQL*Loader is used to load a group of text files as external tables to the database. In the analysis phase, 22 complex analytical SQL query statements are executed in sequence. In the update phase, a group of existing data is deleted from the database. From the perspective of I/O layer, the test model is one of the most typical OLAP business models, where large blocks are accessed in sequence by multiple streams and the ratio between reads and writes is 9: Acronyms and Abbreviations Table 1-1 Acronyms and abbreviations Acronym and Abbreviation AWR OLAP OLTP PGA Full Name Automatic Workload Repository Online Analytical Processing Online Transaction Processing Progress Global Area Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 5

6 Acronym and Abbreviation SGA ETL Full Name System Global Area Extract, Transform, Load Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 6

7 2 Products and Technologies 2.1 OceanStor V3 Converged Storage Systems HUAWEI OceanStor V3 converged storage systems are next-generation unified storage products designed for enterprise-class applications. Leveraging a storage operating system oriented to cloud architecture, a powerful next-generation hardware platform, and a full range of intelligent management software, OceanStor V3 converged storage systems deliver industry-leading functionality, performance, efficiency, reliability, and ease of use. They provide data storage for applications such as large-scale database OLTP/OLAP, file sharing, and cloud computing, and can be used in industries ranging from government, finance, telecommunications, energy, to media and entertainment (M&E). Meanwhile, OceanStor V3 converged storage systems can provide a wide range of efficient and flexible backup and disaster recovery solutions to ensure business continuity and data security, delivering excellent storage services. For details about HUAWEI OceanStor V3 converged storage systems, click the following link: Figure 2-1 HUAWEI OceanStor V3 converged storage systems Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 7

8 2.1.1 Next-Generation Hardware OceanStor V3 converged storage systems employ next-generation Intel multi-core processors, PCIe 3.0 buses, 12 Gbit/s SAS 3.0 high-speed disk ports, and a variety of host ports such as 16 Gbit/s Fibre Channel, 10 Gbit/s FCoE, and 56 Gbit/s InfiniBand host ports. The storage systems provide up to 28 GB/s of system bandwidth to meet the requirements of bandwidth-intensive application scenarios. They also offer million-level IOPS performance, outshining products from other vendors. OceanStor V3 converged storage systems are equipped with exclusive SmartIO cards. A SmartIO card supports 8 Gbit/s Fibre Channel, 16 Gbit/s Fibre Channel, 10 Gbit/s iscsi, and 10 Gbit/s FCoE. Users can specify the protocols that a SmartIO card is required to support. The deduplication/compression cards used by OceanStor V3 converged storage systems support lossless data deduplication and compression, efficiently reducing data storage costs. In addition, the storage systems can implement data encryption to secure data Multi-Controller Architecture The multi-controller architecture used by OceanStor V3 converged storage systems supports online horizontal expansion. An OceanStor V3 converged storage system can be non-disruptively expanded to a maximum of eight controllers, 1 TB of cache, and 5 TB of storage space, meeting customers' future capacity needs. The multi-controller architecture allows load balancing among controllers and eliminates single points of failure, thereby ensuring high availability and stable service running Convergence Design Smart Software Convergence of SAN and NAS: SAN and NAS services are converged to provide elastic storage, simplify service deployment, improve storage resource utilization, and reduce the total cost of ownership (TCO). Underlying storage resource pools directly provide both block and file services, thereby shortening storage resource access paths to ensure that the two services are equally efficient. Convergence of heterogeneous storage systems: Based on the built-in heterogeneous virtualization function, OceanStor V3 converged storage systems can efficiently manage storage systems from other mainstream vendors and unify resource pools for central and flexible resource allocation. Convergence of entry-level, mid-range, and high-end storage systems: OceanStor V3 converged storage systems are the only storage systems in the industry that enable entry-level, mid-range, and high-end storage systems to interwork seamlessly with each other. Data can freely flow among storage products of different models without the assistance of third-party systems. Convergence of SSDs and HDDs: The advantages of traditional and solid-state storage media are combined, bringing the performance of different types of storage media into full play and striking an optimal balance between performance and cost. Convergence of primary and backup storage: The built-in backup function enables data to be efficiently backed up without additional backup software, simplifying backup solution management. Multi-tenancy and service level agreement (SLA): OceanStor V3 converged storage systems intelligently allocate storage resources in cloud computing environments to meet the needs of enterprises and organizations. The storage systems also leverage data Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 8

9 isolation and a range of data security policies such as data encryption and data destruction to meet varying data security requirements. OceanStor V3 converged storage systems provide four service levels and allocate storage resources based on service priorities. Storage resources are first allocated to high-priority services to ensure system performance and shorten response time. Smart-series efficiency improvement suite: OceanStor V3 converged storage systems use SmartTier (dynamic storage tiering), SmartMotion (intelligent data migration), and innovative SmartVirtualization (heterogeneous virtualization) to achieve vertical, horizontal, and cross-system 3D data flowing, improving storage resource utilization by three times. Hyper-series data protection software: Data protection software such as remote replication, snapshot, and LUN copy software meets user needs for local, remote, and multi-region data protection, maximizing business continuity and data availability Unified and Easy Management Unified management: OceanStor V3 converged storage systems provide powerful storage management software that supports global topology view, capacity analysis, performance analysis, fault diagnosis, and end-to-end service visualization to manage a wide range of devices. Convenient management: OceanStor V3 converged storage systems can be initially configured in 5 steps which take about 40 seconds, and expanded in two steps which take about 15 seconds. Mobile management: Users can use tablets and mobile phones to manage storage systems in real time. System status is sent automatically, making constant attendance by an engineer unnecessary RAID 2.0+ Block Virtualization RAID 2.0+ block virtualization of the OceanStor V3 implements virtualization for underlying disk management and upper-layer resource management. Inside the system, the storage space of each disk is divided into fine-grained data blocks, which comprise RAID groups. In doing so, data is evenly distributed to all disks in the storage pool. In addition, data block based resource management largely improves the resource management efficiency. 1. The V3 storage systems support SSDs, SAS disks, and NL-SAS disks. These disks comprise disk domains. In a disk domain, disks of the same type comprise disk groups (DGs). 2. In a DG, the storage space of disks is divided into chunks (CKs) of a fixed size. Then the system consolidates CKs from random disks into CK groups (CKGs) based on RAID algorithms. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 9

10 3. CKGs are divided into logical storage space called extents, which have a fixed size too. Extents are the minimum unit for comprising thick LUNs. On thin LUNs, extents are further divided into smaller grains. 2.2 Oracle Database and Cluster Oracle Database is one of the most widely used relational databases. This section briefly introduces Oracle Database 12c and focuses on Multitenant-related components and features, including RAC, ASM, Multitenant, data files, database instance architecture, and application types Oracle RAC and ASM As shown in Figure 2-2, an Oracle 12c RAC contains two types of nodes: Hub nodes and Leaf nodes. Hub nodes have direct access to shared storage, whereas Leaf nodes access shared storage through Hub nodes. When a database is deployed on an Oracle RAC, the nodes can be grouped into multiple server pools. Each database is deployed in a server pool, and every node in a server pool runs a database instance. Application servers access the virtual IP addresses (VIPs) of nodes to store data. If a node fails, its VIP network is restored on another node of Oracle RAC. Application servers reconnect to the Oracle database through a reconnection mechanism. Setting connection character strings on application servers can enable multiple modes of accessing Oracle RAC nodes, including load balancing and failover modes. In these modes, a multi-node Oracle cluster is presented as a single database to application servers. The shared storage of the Oracle RAC Hub nodes includes Oracle Cluster Registry (OCR), voting disks, and database. OCR records information about node statuses, voting disks synchronize data between nodes, and the database is a set of files. Figure 2-2 Oracle Flex Cluster 12c Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 10

11 Oracle ASM provides a simple storage management interface for database administrators to manage servers and storage across different platforms. As a built-in file system and volume manager, Oracle ASM is exclusive to Oracle database files. ASM simplifies file system management, provides asynchronous I/O performance tuning, saves management time for administrators, and offers a flexible, efficient database environment. ASM can consolidate LUNs into a disk group and use Allocation Units (AUs) to allocate storage space from the disk group. ASM supports three types of disk groups. External: Data is not mirrored between LUNs, and the storage system provides data protection. Normal: A normal disk group consists of two failure groups between which data is mirrored. High: A high disk group consists of three failure groups among which data is mirrored. When an OceanStor V3 storage system is used to create ASM disk groups, it is recommended that controllers A and B evenly own the LUNs in the disk groups before external or normal disk groups are created. Oracle Flex ASM is a new Oracle ASM deployment model that increases database instance availability and reduces Oracle ASM related resource consumption. Oracle Flex ASM facilitates cluster based database consolidation, as it ensures that Oracle Database 12c instances running on a particular server will continue to operate, should the Oracle Flex ASM instance on that server fail. Figure 2-3 ASM before Oracle Database 12c Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 11

12 Figure 2-4 Oracle 12c Flex ASM Oracle RAC provides the following key characteristics, essential for HA data management: Reliability Oracle Database is known for its reliability. Oracle RAC takes this step further by removing the database server as a single point of failure. If an instance fails, the remaining instances in the cluster remain open and active. Oracle Clusterware monitors all Oracle processes and immediate restarts any failed component. Error detection Oracle Clusterware automatically monitors all Oracle RAC databases as well as other Oracle processes (Oracle ASM, instances, Listeners, etc.) and provides fast detection of problems in the environment. It also automatically recovers from failures often before users notice that a failure has occurred. Recoverability The Oracle Database includes many features that make it easy to recover from various types of failures. If an instance fails in an Oracle RAC database, it is recognized by another instance in the cluster and recovery will start automatically. Fast Application Notification (FAN) and Fast Connection Failover (FCF) and especially the Oracle RAC 12c Application Continuity feature make it easy to mask any component failure from the user. Continuous Operations Oracle RAC provides continuous service for both planned and unplanned outages. If a server (or an instance) fails, the database remains open and applications continue to be able to access data, allowing for business critical workloads to finish, mostly without a delay in service delivery. For more information about Oracle 12c RAC and ASM, refer to the following documents: White Paper: Oracle Real Application Clusters (RAC) A Technical Overview of New Features for Automatic Storage Management in Oracle Database 12c Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 12

13 2.2.2 Oracle System Architecture Figure 2-5 Oracle system architecture As shown in the preceding figure, System Global Area (SGA) and Program Global Area (PGA) of Oracle databases consume memory. SGA stores system information and page cache information, and PGA stores session information. SGA contains the following parts: Buffer Cache: buffers data blocks. Redo Log Buffer: buffers log records as a recycle data group. Share Pool: buffers data dictionaries and shared SQL information. Oracle files are categorized as follows: Control file: records the database structure, parameters, and locations of other data fails. Data file: stores user data and temporary data. Online log: record changes to data blocks and consists of several log groups. Files in the log groups are mirrored to each other. After a log group is used up, data in written to the next log group. After the first log group is used up, data is written to the first log group again. Archive log: In archive mode, databases copy fully written log groups to the archive area for data restoration when anomalies occur. Among Oracle processes, the Listener process monitors client connections. Clients are connected in two modes: In shared mode, the listener redirects client requests to the dispatcher process, which places the request in the request queue of the large pool. Then the shared server obtains and processes the request in the request queue and puts the processing results in the Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 13

14 response queue of the large pool. At last, the dispatcher returns the processing results to the client. In dedicated mode, a dedicated server process serves each client connection. After receiving the request from a client, the Oracle server process looks for the data block in the buffer cache. If the data block is found, data is read, computed, and changed in the buffer cache. If the data block does not exist in the buffer cache, the Oracle server process writes the data block from data files to the buffer cache, and then computes and changes it. Oracle uses the LRU algorithm to eliminate outdated data in the buffer cache so that the released storage space can be used by new data blocks. Data that has been changed in the buffer cache is called "dirty data", which is written by the DB writer (DBW) process to data files. To ensure data integrity and reliability, relational databases use "transaction" to indicate an atomic operation. When processing a transaction, the Oracle server process records changed data and the change time in the log buffer. When the transaction is delivered, the log writer (LGWR) synchronizes the log data in the log buffer to online log files. The log buffer is a memory area where data can be written in a circular manner. When the log buffer is one-third full, the LGWR synchronizes the log records to the log file, regardless of whether the transaction is committed. In addition, Oracle databases synchronize logs automatically every three minutes. By default, Oracle databases perform a checkpoint every 30 minutes. When the checkpoint is performed, the Checkpoint (CKPT) process synchronizes the control file and triggers the DBW to write dirty data to data files. Online logs of Oracle are divided into several groups, each of which contains one or multiple log files. When multiple log files exist, the files are mirrored to each other. Oracle databases write logs to the log groups in sequence. When the last log group is filled, the databases write to the first log group, restarting the cycle. Before a log group switch, Oracle checks whether the dirty data recorded in the next log group is completely written to data files. If not, Oracle waits until the DBW process writes all the dirty data to the data files before it starts the log group switch. When an Oracle database is in archive mode, the Archive (ARC) process copies filled logs to the archive area. If data anomalies occur, the archived logs are used for precise data recovery. Oracle Database 12c also supports a Multitenant that allows multiple PDBs to run in one CDB. Figure 2-6 shows a CDB with four containers: the root, seed, and two PDBs (hrpdb and salespdb). Each PDB has its own dedicated application. A different PDB administrator manages each PDB. User SYS can manage the root and every PDB. At the physical level, this CDB has a database instance and database files. The Multitenant feature brings better serviceability to the Oracle Database. The seed PDB is a system-supplied template that the CDB can use to create new PDBs. The seed PDB is named PDB$SEED. You cannot add or modify objects in PDB$SEED. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 14

15 Figure 2-6 Oracle Multitenant architecture For more information about Oracle Multitenant Architecture, refer to the following document: White Paper: Oracle Multitenant Oracle Application Types Data transactions are categorized as two types: OLTP and OLAP. OLTP: A number of online users perform transactions. OLAP: A small number of users perform long-term complex statistical queries. OLAP applications have the following I/O characteristics: From the perspective of database administrators No data or only a small amount of data is modified. Complex query statements are invoked, and a large number of lines are scanned. A query takes hours or even days, depending on the complexity of query statements. Data aging can be measured by hour or day. The query output is typically a statistical value, obtained using group by and order by. From the perspective of storage sampling Every I/O is large-sized, ranging from 64 KB to 1 MB. Data is read in sequence. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 15

16 When read operations are being performed, write operations are performed in a temporary tablespace if any. Online logs are seldom written unless data is loaded in batches. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 16

17 3 Best Practices for Oracle Database Planning and Configuration This chapter describes the recommended plan and configuration, including networking, storage plan and configuration, as well as host and database parameter settings, for the scenario where OLAP Oracle Database 12c is deployed based on an OceanStor V3 converged storage system. 3.1 SAN Networking Oracle databases typically serve enterprises' mission-critical application systems. To ensure business continuity, it is recommended that an Oracle RAC cluster be used. The network must prevent single points of failure. As shown in the following figure, the SAN network uses two switching planes that are physically isolated from each other. Each switching plane consists of one switch or multiple interconnected switches. Each of the database nodes and storage controllers is connected to both switching planes. Figure 3-1 SAN network Application layer Management network Public network RAC interconnection network... Oracle RAC Storage Area Network OceanStor 6800 V3 Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 17

18 3.1.1 Setting Zones and vlans Zones and vlans can be set to serve the following purposes: In the IT environment of an enterprise, a storage system typically serves multiple application systems. Zones or vlans can be set to prevent application systems from affecting each other. An excessive number of host paths will complicate system management and make network adjustment more difficult during the O&M process. Zones or vlans can be set to improve IT infrastructure flexibility. The overall policy for setting zones or vlans is to ensure dual-switching networking. That is, on each switching plane, each of the database nodes and storage controllers has at least one logical channel. Based on this policy, consider networking from the following aspects: performance, deployment complexity, and O&M flexibility. In terms of performance, plan the number of host ports and paths based on the IOPS and bandwidth requirements of hosts. In terms of deployment and O&M, strike a balance between deployment complexity and O&M flexibility. The following figure shows three zoning approaches. Zoning by application has the lowest complexity and flexibility, zoning by host has the medium, and zoning by port has the highest. Figure 3-2 Zoning or vlan setting granularity Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 18

19 3.2 Storage Configuration Figure 3-3 Resource allocation flowchart Planning Before using an OceanStor V3 converged storage system, it is recommended that you analyze application performance and capacity requirements and plan the number of disks so that you can minimize O&M risks. In addition to a series of entry-level, mid-range, and high-end storage systems, Huawei provides solid state storage that features low latency and high performance. In terms of deploying Oracle Database, analysis of storage performance and capacity requirements is necessary to selection of storage systems and disks. In storage performance assessment, performance factors are typically first considered and then capacity factors. In OLAP Oracle Database applications, I/O characteristics are multiple channels of sequential reads. The major storage performance indicator is the amount of data processed per second, namely, the bandwidth. Based on the following parameter values and formulas, you can calculate the performance of RAID levels. Then, estimate the required number of disks. Table 3-1 Assessment of OLAP application performance RAID Level OLAP Throughput (MB/s) For Example, 2 to 8 Gbit/s FC RAID Group Comprised of Eight 10k rpm SAS Disks RAID 10 MIN (MBPS PATH, MBPS DISK x N) MIN (1350, 50 x 8) = 400 RAID 5 MIN (MBPS PATH, MBPS DISK x N) MIN (1350, 50 x 8) = 400 RAID 6 MIN (MBPS PATH, MBPS DISK x N) MIN (1350, 50 x 8) = 400 MBPS DISK indicates the multi-stream sequential read throughput of a single disk. For OceanStor V3 converged storage systems, MBPS DISK of a 10k rpm HDD is 50, and that of a 7.2k rpm HDD is 30. N indicates the number of disks that compose a RAID group. MBPS PATH indicates the total bandwidth provided for the connections between all database hosts and the storage system. In the formulas, the maximum bandwidth of a single 16 Gbit/s Fibre Channel link is 1350 MB/s. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 19

20 3.2.2 Disk Domains Capacity estimation covers user tablespaces, online logs, archive logs, and backup sets. User tablespaces and other database areas are typically stored in different LUNs for high performance and reliability. In OLAP applications, online analysis is implemented and most I/Os are read I/Os. In capacity estimation, the amount of data in user tablespaces and that in temporary tablespaces are the major considerations. If archived data and backup data will be stored, extra capacity must be considered too. Typically, the temporary tablespace size should be set to the same as the user tablespace size, so that there is sufficient space for sorting operations. The use of RAID 2.0+ makes it difficult to estimate the number of disks based on application performance and capacity requirements. Huawei UniSTAR edesigner ( which is a solution design tool, includes two assessment tools oriented to storage systems. It is recommended that customers use the two tools to plan the number of disks for Oracle Database. Product capability assessment: Based on a specific storage configuration, the performance and capacity that can be provided by a storage system are assessed. Storage configuration wizard: Based on specific application requirements, the storage model and disk configuration are assessed. Huawei provides verified data warehouse reference architectures based on OceanStor 5300 V3, 5500 V3, 5600 V3, and 5800 V3 converged storage systems. When designing a storage solution, you can refer to HUAWEI OceanStor V3 Converged Storage Systems OLAP Oracle Database Reference Architectures. A disk domain is a combination of multiple disks. After disks are consolidated and a certain amount of hot spare capacity is reserved, a disk domain provides storage resources for storage pools in a unified manner. One or more disk domains can be created in an OceanStor V3 converged storage system. Multiple storage pools can be created in a disk domain. A disk domain can consist of SSDs, SAS disks, and/or NL-SAS disks. Disk domains are isolated from each other, including performance, storage resources, and faults. The percentage of hot space capacity in a disk domain is related to the hot spare policy and the number and types of disks. The percentage is automatically maintained by the storage system based on a reliability engineering method. If you need to plan storage capacity and performance, it is recommended that you use Huawei UniSTAR edesigner. In terms of disk domain creation, the default hot spare policy is high. After a disk domain is created, you can also change the policy to low or none based on reliability requirements of applications. However, considering reliability, it is recommended that you do not change the hot spare policy, especially not to change the policy to none. OceanStor V3 converged storage systems support SSDs, SAS disks, and NL-SAS disks and provide tiered storage. For OLAP Oracle Database, it is recommended that you use SSDs or SAS disks, and do not use NL-SAS disks that provide low performance and reliability. OLAP typically involves complex queries. A large number of data blocks are accessed, whereas the access frequency is low. No obvious hotspot data exists. Therefore, it is recommended that you do not use SmartTier for hotspot data identification and tiered storage. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 20

21 3.2.3 Storage Pools A disk domain in an OceanStor V3 converged storage system can consist of over 1000 disks. However, considering performance and reliability, it is recommended that each storage tier of a disk domain contain 100 disks at most. A storage pool, a container that stores storage space resources, is created in a disk domain. A storage pool can dynamically allocate resources from a disk domain and define the RAID level of each storage tier. A storage tier is a collection of storage media providing the same level of performance in a storage pool. Different storage tiers manage storage media with different performance levels and provide different storage spaces for applications whose performance requirements vary. There are three types of storage tiers: high-performance tier that consists of SSDs (including SLC and emlc), performance tier that consists of SAS disks (including 15k rpm and 10k rpm), and capacity tier that consists of NL-SAS disks. OceanStor V3 converged storage systems support seven RAID levels: RAID 6, RAID 10, RAID 5, RAID 3, RAID 50, RAID 1, and RAID 0. The most commonly used RAID levels are RAID 6, RAID 10, and RAID 5. From the perspective of stripe width, RAID 6 and RAID 5 are classified into RAID 6-4 (2D2P), RAID 6-6 (4D2P), RAID 6-10 (8D2P), RAID 5-3 (2D1P), RAID 5-5 (4D1P), and RAID 5-9 (8D1P). It is recommended that you set a RAID level for OLAP Oracle Database based on the following rules: RAID 6 is preferred, especially for mission-critical database applications that require high reliability. If capacity takes precedence over reliability, RAID 5 is recommended. If a capacity tier is configured for OLAP Oracle Database, it is strongly recommended that RAID 6 be set for the capacity tier. OceanStor V3 converged storage systems employ innovative RAID When creating a storage pool, you can set the extent size. An extent consists of one or more chunk groups (namely, RAID stripes). Extent is the basic unit used by a storage pool for allocating space to a LUN. The default extent size is 4 MB, which is applicable to most scenarios. Extent is also the basic unit of block virtualization. Functions such as hotspot data identification and migration and data motion are implemented based on extents too. It is recommended that you set the extent size for OLAP Oracle Database based on the following rules: 4 MB is preferred. If you want to improve the speed of data motion, it is recommended that you set the extent size to a larger value. During the creation of a storage pool, you can set an alarm threshold for the capacity allocation ratio. The default threshold is 80%. Capacity alarming is particularly important in scenarios where value-added features such as thin LUN, snapshot, remote replication, and clone are used. You can set a proper alarm threshold based on the speed of application data growth to prevent insufficient capacity of the storage pool from causing application interruption. When creating a storage pool in developer mode, you are allowed to set a stripe depth. For Oracle OLAP applications, it is recommended that you set a large stripe depth value (for example, 512 KB) because the application type is large data block reading, where data is not frequently written and the amount of written data is small. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 21

22 3.2.4 LUNs In terms of LUNs created for OLAP Oracle Database, it is recommended that you set their major parameters based on the following rules: Owner controller (owner_controller): If the database workload is heavy, it is recommended that you evenly allocate LUNs to controllers to ensure load balancing among these controllers. Capacity allocation policy (allocation_policy): Four capacity allocation policies are provided: Automatic allocation, Allocate from the high performance tier first, Allocate from the performance tier first, and Allocate from the capacity tier first. This parameter is used with tier storage. Typically, it is recommended that you set the policy to Automatic allocation for Oracle storage. If you want to accelerate access to an identified hotspot area, you can set the policy to Allocate from the high performance tier first for that area. If you do not want archived data or backup data to consume the space of the performance tier, you can set the policy to Allocate from the capacity tier first. Read cache policy (read_cache_policy): OLAP applications are characterized by sequential workload. If an OLAP application shares storage with other applications, the reclamation policy is recommended. Write cache policy (write_cache_policy): OLAP applications are characterized by sequential workload. If an OLAP application shares storage with other applications, the reclamation policy is recommended. Prefetch policy (prefetch_policy): Four prefetch policies are provided: non-prefetch, constant prefetch, variable prefetch, and intelligent prefetch. For details about the four prefetch policies, see the OceanStor V3 converged storage system user manual. OLAP Oracle Database is characterized by multiple channels of sequential large I/Os. Therefore, it is recommended that you use the non-prefetch policy if the number of concurrent accesses is large, and use the intelligent prefetch policy if the number of concurrent accesses is small. I/O priority (io_priority): Three levels are provided. The default level is Middle. It is recommended that you set the I/O priority of the log area to High and that of the archive area to Low. LUN type (lun_type): Two types are provided: Thick and Thin. If performance takes precedence over capacity (for example, a mission-critical production system), it is recommended that you set the LUN type to Thick. If capacity takes precedence over performance (for example, a testing/development system), it is recommended that you set the LUN type to Thin. Thin is not recommended for scenarios where a high write bandwidth is required, such as a backup target or temporary tablespace area. LUN capacity and quantity Different from a traditional RAID group that contains 10+ disks at most, a storage pool that employs the RAID 2.0+ mechanism contains LUNs across all disks in the disk domain. The number of disks in a disk domain can be more than 100. To bring disk performance into full play, it is recommended that you configure LUN capacity and quantity based on the following rules: Total number of LUNs in a disk domain: Not less than the number of disks x 4 32 is recommended. (4 is the proper number of concurrent access requests per disk, and 32 is the default maximum queue depth of a LUN). LUN capacity: If the preceding condition is met, prefer large-sized LUNs to simplify management overheads. In terms of the maximum capacity of a LUN, you must consider Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 22

23 3.2.5 Mapping Views the restrictions posed by the operating system and Oracle Database. For example, Oracle 11g ASM requires that the capacity of a LUN do not exceed 2 TB. A mapping view defines a logical mapping among LUNs, array ports, and host ports. It is recommended that you create a mapping view based on the following rules: A LUN group is an object designed to facilitate LUN resource management. Typically, LUNs that serve the same application should be added to one LUN group, for example, LUNs that belong to each storage area of Oracle RAC. A host group is a collection of hosts that need to share storage resources. Each host contains multiple initiators (host ports). It is recommended that you create a host for each server and add all initiators of a server to the host created for that server. Port groups help users allocate storage ports in a more fine-grained manner. Port groups are not necessary. However, it is recommended that you allocate a port group for an OLAP application to improve O&M flexibility and reduce performance impacts between applications. To prevent single points of failure, each port group must contain at least one port on each controller. 3.3 Host Configuration Queue Depth I/O queue depth is an important parameter that determines the performance of Oracle Database. Two parameters on the operating system layer affect the I/O queue depth: block device's queue depth and HBA's queue depth. It is recommended that you set a block device's queue depth and HBA's queue depth based on the following rules: In a Linux operating system, the block device's maximum queue depth is 128. It is recommended that you do not change the value. An HBA's queue parameters vary depending on the HBA type and driver. For details, see the specifications provided by the HBA vendor. For example, the QLogic 8 Gbit/s dual-port Fibre Channel HBA allows the maximum queue depth of each LUN to be 32. It is recommended that you do not change the maximum queue depth of an HBA. To increase the overall I/O queue depth, it is recommended that you increase the number of LUNs. For an AIX operating system, it is recommended that you install UltraPath and do not use the multipathing software delivered with the operating system or provided by a third party. After UltraPath is installed, the block device's maximum queue depth is changed to 32. It is recommended that you do not change the value. If UltraPath is not installed, the block device's default maximum queue depth is 5. It is strongly recommended that you change the value to 32 or larger. In an AIX operating system, the HBA's default maximum queue depth is 200. You can change the value based on application requirements. For a Windows operating system, the maximum I/O queue depth of a LUN is also determined by the specifications provided by the HBA vendor. It is recommended that you do not change the value. For other operating systems, adjust the queue depth as instructed in the user manual of that operating system. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 23

24 In AIX, you can run lsattr -El fcs0 to query the parameter settings of HBA fcs0. See the following figure. num_cmd_elems indicates the maximum I/O queue depth of fcs0, and the default value is 200. You can run chdev -Pl fcs0 -a num_cmd_elems=512 to change the maximum I/O queue depth of fcs0 to 512. In AIX, you can run lsattr -El hdisk1 to query the maximum I/O limit (max_transfer) and maximum queue depth (queue_depth) of hdisk1. You can run chdev -Pl hdisk1 -a queue_depth=32 to change the maximum queue depth of hdisk1 to I/O Alignment If MBR partitions are created in Linux or Windows whose version is earlier than Windows Server 2003, the first 63 sectors of a disk are reserved for the master boot record and partition table. The first partition starts from the 64th sector by default. As a result, misalignment occurs between data blocks (database or file system) delivered by hosts and data blocks stored in the disk array, causing poor I/O processing efficiency. When creating MBR partitions in Linux, it is recommended that you enter the expert mode of the fdisk command and set the start location of the first partition to the start location of the second extent on a LUN. (The default extent size is 4 MB.) The following is a quick command used to create an MBR partition in /dev/sdb. The partition uses all space of /dev/sdb. The start sector is set to 8192, namely, 4 MB. printf "n\np\n1\n\n\nx\nb\n1\n 8192\nw\n" fdisk /dev/sdb Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 24

25 The step-by-step command used to set partition alignment is described as follows: fdisk n p /dev/sdb (Create a partition.) (Set the partition type to maser.) 1 (Set this partition as the first partition.) x b (Press Enter. Set the start location of the partition to the default start sector, namely, 64.) (Press Enter. Set the end location of the partition to the default last sector.) (Enter the expert mode.) (Set the start location of the partition.) 1 (Set the first partition.) 8192 (Set the start location to 8192, namely, 4 MB.) w (Write data to disks and quit.) Another way to resolve partition misalignment in Linux is to use GPT partitions. The following is a quick command used to create a GPT partition in /dev/sdb. The partition uses all space of /dev/sdb. The start sector is set to 8192, namely, 4 MB. parted -s -- /dev/sdb "mklabel gpt" "unit s" "mkpart primary " "print" To create MBR partitions in Windows whose version is earlier than Windows Server 2003, it is recommended that you run diskpart to set partition alignment. diskpart> select disk 1 diskpart> create partition primary align= Block Device Scheduling Algorithms Linux 2.6 kernel supports four types of block device scheduling algorithms: noop, anticipatory, deadline, and cfq. In OLAP application scenarios, if traditional disks are used, the deadline algorithm is recommended; if SSDs are used, the noop algorithm is recommended. Change the configuration file /boot/grub/menu.lst and add elevator=algorithm name in the kernel line. This method applies to all block devices. kernel /vmlinuz el5 ro root=/dev/volgroup00/logvol00 rhgb quiet elevator=deadline Block Device I/O Settings In Linux, max_sectors_kb, a block device parameter, restricts the maximum I/O data block size. Set max_sectors_kb to the database I/O size. For example, the typical data block size used by an OLAP database is 32 KB. The db_file_multiblock_read_count parameter specifies the number of concurrent I/O blocks. If db_file_multiblock_read_count is set to 32, the concurrent I/O size of the database is 1024 KB. In this case, it is recommended that you change the value of max_sectors_kb to In the configuration file /etc/rc.local, you can add the following command that enables the value of max_sectors_kb to be changed to 1024 upon each startup. echo 1024 > /sys/block/sdb/queue/max_sectors_kb Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 25

26 3.4 Database Configuration Database Parameters Database Block Size The block size used by Oracle Database ranges from 2 KB to 32 KB, with the default value of 8 KB. Users can change the block size as required. For a typical OLTP application, the database block size is usually set to the default value 8 KB. However, for an OLAP application, it is recommended that the database block size be set to a larger value, so that the application performance is significantly improved. Memory Allocation The db_block_size parameter indicates the database block size. You need to set it when creating a database. After that, you cannot change the value. In Oracle database applications, the memory should be fully used without compromising the normal system running. It is recommended that 80% of the physical host memory be allocated to Oracle Database. In OLAP applications, a large number of sorting operations are performed. When dedicated servers are used to access the database, sorting operations occur in the PGA. It is recommended that at least 50% of the memory allocated to Oracle Database be allocated to the PGA for improving system performance. Parallel Operations Assume that MEM indicates the physical memory of a host. Set database instance parameters memory_target and memory_max_target to 0. Set sga_max_size and sga_target to 50% x MEM. Set pga_aggregate_target to 50% x MEM. OLAP Oracle Database involves a large number of complex query and analytical statements, with the possibility of massive full-table scanning operations. If the host is equipped with multi-core CPUs, parallel queries can greatly improve the query performance but consume a lot of CPU resources of the host. It should be noted that, the maximum number of parallel queries is related to the concurrency capabilities of tables. I/O Policy Assume that the number of CPUs in a host is CPU_COUNT and the number of cores on each CPU is PARALLEL_THREADS_PER_CPU. Then, you can set parallel_max_servers to PARALLEL_THREADS_PER_CPU x CPU_COUNT x 4 x 5, indicating that a maximum of parallel_max_servers parallel operations can be concurrently performed. Set parallel_min_servers to a value that ranges from 0 to parallel_max_servers, indicating the number of processes concurrently running after the database instance starts. If a database is deployed in file system mode, the I/O policy can be set to synchronous I/O, asynchronous I/O, direct I/O, and/or merged I/O. To achieve a better I/O efficiency, it is recommended that you enable both asynchronous I/O and direct I/O. Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 26