ARCHITECTING COST-EFFECTIVE, SCALABLE ORACLE DATA WAREHOUSES White Paper May 2009 Abstract This paper describes reference configuration and sizing information for Oracle data warehouses on Sun servers for small, medium, and large solutions to meet business intelligence/data warehouse needs.
Table of Contents Introduction...1 Solution... 1 Balanced configurations... 2 Protecting investments... 2 Workloads... 3 Servers and storage... 3 Reference Configuration Examples...4 Small, up to 5.76 TB... 4 Small, up to 17.92 TB... 5 Medium, up to 35.84 TB... 5 Large, up to 143.36 TB... 6 Partial reference configuration list... 7 Reliability and high availability... 8 Oracle Optimized Warehouse for Sun... 9 Summary... 10 Getting started... 10
1 Architecting Cost-Effective, Scalable Oracle Data Warehouses Chapter 1 Introduction Today, companies rely on data warehouses to constantly and continuously make more and better informed decisions. As the importance of databases to the business grows, so does the requirement for higher availability and reliability. In addition, with the ability to make smarter decisions comes the desire to speed the process in order to beat the competition and cut costs wherever possible. This requirement to deliver real-time data to users requires more performance and throughput. Compounding these issues is the reality that data doubles every 12 to 18 months fueled by new data types such as images, video, voice, and radio frequency identification (RFID) chips requiring platforms that can scale easily and costeffectively. Finally, the fact that data warehouse environments are constantly changing and growing points to an obvious requirement for flexibility and adaptability. Architecting cost-effective, fast, reliable, and scalable data warehouse environments can be a time-consuming task. To stay on top of the game, IT managers need costeffective, powerful solutions that can be quickly implemented to provide value and a competitive edge to the business. Leveraging vast engineering and field experience architecting and implementing data warehouses with Oracle Database, Sun created a set of data warehouse reference configurations to help IT managers quickly choose a configuration to meet their needs, thus reducing the risk and deployment times for data warehouse solutions. Solution Sun s end-to-end, innovative solutions software, servers, storage, and service provide all of the technology requirements (performance, scalability, availability, and security) for complete business intelligence data warehouse (BIDW) implementations. Although Sun can address every aspect of a BIDW environment in terms of infrastructure, the suggested reference configurations in this paper focus on the data warehouse. The reference configurations are divided into three groups to highlight the specific needs of different sized implementations: Small for databases with a raw data size up to 25 TB Medium for databases with a raw data size of up to 75 TB Large and extra large for databases with a raw data size over 75 TB
2 Architecting Cost-Effective, Scalable Oracle Data Warehouses The reference configurations in this paper are suggested starting configurations that should be optimized based on specific requirements and preferences for technology. The configurations are based on generalized workloads. Final solutions might need adjustment of CPUs, cores, HBAs, RAID controllers, channels, and/or disk drives to fit the reference configuration to specific requirements. Contact Sun to confirm that the selected configuration meets specific needs or to adjust the parameters to meet those needs. Balanced configurations Each of the reference configurations is designed to eliminate bottlenecks, providing a balance of processing power, throughput, and storage capacity/throughput. The reference configurations are also designed to provide balanced, modular building blocks that can easily scale by adding CPUs, nodes, channels, and drives as the needs of the data warehouse grow. The Solaris 10 Operating System (OS) is scalable to over 100 CPUs and Sun systems and storage can be scaled up (vertically) or out (horizontally) without disturbing the business. A few of the features that enable Sun servers to balance processing and throughput are CMT, or chip multithreading, and MPO, or memory placement optimization. CMT is the capability to run multiple, simultaneous threads on a single CPU core and is featured in UltraSPARC T1, T2, and T2+ processors. CMT provides several benefits including more throughput and scalability for multithreaded applications, as well as reduced power requirements. MPO works to help ensure that memory pages are as close as possible to the processors that access them, resulting in increased performance. As a result, throughput performance increases. In addition to improving individual job performance, it also increases overall system throughput for multiple applications. Protecting investments Data warehouses require a long-term foundation that can encompass new technologies as they evolve without massively disrupting operations. Systems and software based on open standards can more readily absorb change and be redeployed than closed proprietary systems. Sun embodies this principle by maintaining binary compatibility between operating system releases, enabling existing Solaris applications to run unmodified on the Solaris 10 OS. This means that Solaris applications developed 10 years ago can run on the Solaris 10 OS unchanged, taking advantage of new and advanced Solaris OS features. With the Solaris 10 OS, the same features and services run on UltraSPARC, AMD Opteron, and Intel Xeon processor-based systems. In fact, Sun guarantees source code compatibility between SPARC and x64 processors, enabling applications to run across platforms with a simple recompile. Sun s binary compatibility makes it easier to re-purpose systems to other tasks if necessary because the same applications run across the entire Sun product line.
3 Architecting Cost-Effective, Scalable Oracle Data Warehouses Furthermore, Sun storage maintains high levels of heterogeneous interoperability with leading platform and OS vendors, offering investment flexibility and protection now and in the future. Workloads The reference configurations are designed to support the three most common workloads observed by Sun engineers but are mainly designed for heavy workloads. Light workload this type of workload is characterized by light users who perform straight selects from well-indexed and/or partitioned tables. Typical jobs include no more than two-way joins or display of pre-calculated reports. Standard workload supporting users who perform three-way or five-way joins, moderate sorts, and a few full table scans on moderate-sized tables. Heavy workload typified by users who perform complex sorts, six-way joins (or more), and full-table scans of moderate to large tables. Servers and storage Configurations for each database size utilize both vertical and horizontal scalability, as well as different CPU architectures for optimum choice and flexibility. Vertical scalability is accomplished by non-disruptively adding CPU and memory into the system, rather than adding entire nodes. The other option for deploying Oracle data warehouses is to use a horizontally scalable model, where scalability is accomplished by adding individual nodes to the database cluster. All configurations are sized to support heavy workloads. The total front-end and back-end bandwidth for the storage subsystem is matched to the expected bandwidth needs of the servers. Sun s broad portfolio of modular and system storage with 4 Gb/sec technology enables optimal configurations of bandwidth, capacity, and cost for horizontal and vertical storage architectures. The configurations utilize 146 and 300 GB 15K RPM drives to provide sufficient capacity, bandwidth, and I/O rate to meet the stated capacities and performance needs.
4 Architecting Cost-Effective, Scalable Oracle Data Warehouses Chapter 2 Reference Configuration Examples Based on their work in the field and lab, Sun engineers created a matrix with reference configurations for data warehouse databases. For the purposes of these reference configurations, raw database size is defined as the raw data without indexes or any other structures. Below are examples for each database size. Small, up to 5.76 TB In this example, Company A requires high availability and expandability. They need to support 50 average concurrent users generating a standard workload against a data warehouse of up to 5 TB, with the capacity to support heavy workloads and growth. Figure 1 shows a suggested configuration of one Sun Fire X4450 servers and two Sun StorageTek 2540 arrays. Powered by dual-, quad- or six-core Intel Xeon processors, and up to 128 GB of memory, the 2- to 24-way Sun Fire X4450 server provides costeffective performance and scalability. This configuration can scale to 23.04 TB by adding the same type of servers and storage. One Sun Fire X4450 server 2 CPUs 32 GB RAM 4x HBA ports SAN Two Sun StorageTek 2540 arrays 72 disks 8x 4 Gb/sec. FC Figure 1. Small data warehouse up to 5.76 TB The Sun StorageTek 2450 array is ideal for server clustering environments. By utilizing two dual-active RAID controllers with mirrored cache, redundant components, and automated I/O path failover, these arrays are well suited where continuous application and data availability are critical. The array also supports direct attachment of multiple hosts or connectivity to a SAN. This makes it prefect for environments that need to initially deploy a direct attached array, and then seamlessly transition to a SAN when ready.
5 Architecting Cost-Effective, Scalable Oracle Data Warehouses Small, up to 17.92 TB Company B needs to implement a medium-sized Oracle data warehouse to support a standard workload and about 16 TB. Figure 2 depicts a configuration using one Sun SPARC Enterprise T5440 server and one Sun StorageTek 6540 array that can scale to 53.76 TB by adding the same servers and storage. One SPARC Enterprise T5440 server 4 CPUs 128 GB RAM 16x HBA ports SAN Sun StorageTek 6540 array 224 disks 16x 4 Gb/sec. FC Figure 2. Small data warehouse up to 17.92 TB The Sun SPARC Enterprise T5440 server is one of the fastest 4-socket servers available with up to 4x higher performance, up to 4x higher performance per watt, and as little as one-fifth the cost of competitive servers. Each processor packs up to eight cores and up to 64 simultaneous threads onto a single piece of silicon for up to 256 threads, together with the key functions of an entire system on a single chip computing, security, and I/O. The Sun StorageTek 6540 array has industry-leading performance, the highest ranking quality in recent customer surveys, and a wide range of data protection options. The array also offers unparalleled flexibility with the ability to mix and match SATA and Fibre Channel drives to meet tiered storage needs and growing backup and archiving requirements. As capacity and data access rate demands increase, more arrays can be added. Medium, up to 35.84 TB Company C needs to quickly install an Oracle data warehouse to support a medium database of 33 TB generating a standard workload. Figure 3 illustrates a configuration using three Sun SPARC Enterprise M5000 servers and two Sun StorageTek 6540 arrays. Based on the advanced dual-core SPARC64 VI or quad-core VII processors with vertical-multithreading (VMT) technology, the Sun
6 Architecting Cost-Effective, Scalable Oracle Data Warehouses SPARC Enterprise M5000 server delivers mainframe-class reliability, scalability, and manageability features to meet the demands of enterprise-class applications such as Oracle data warehouses. SAN Three Sun SPARC Enterprise M5000 servers 8 CPUs 64 GB RAM 8x HBA ports each Two Sun StorageTek 6540 arrays 448 disks 32x 4 Gb/sec. FC Figure 3. Medium data warehouse up to 35.84 TB Powered by up to eight dual-core or quad-core processors, up to 32 cores, up to 256 GB of memory, and with 10 I/O slots, this datacenter system is optimized for data warehouse as well as OLTP databases and other business critical applications. With up to four fault-isolated domains and thousands of Solaris Containers, the Sun SPARC Enterprise M5000 server provides numerous virtualization capabilities for data warehouse consolidation. Plus, the system delivers high levels of reliability and availability through hardware redundancy, Dynamic Reconfiguration, and Solaris Predictive Self Healing. This configuration can be scaled to 71.68 TB by adding the same types of servers and storage. Large, up to 143.36 TB Company D needs to support users generating a standard workload on a large database of over 125 TB. The suggested configuration in Figure 4 uses one Sun SPARC Enterprise M9000 server and two Sun StorageTek 9990V systems for storage. The Sun SPARC Enterprise M9000 server offers the highest availability, highest absolute performance, highest scalability, and the most sophisticated control of resources in Sun s server product line. It supports up to 64 quad-core SPARC64 VII processors, 4 TB of memory, 128 hot-swappable PCI-Express I/O slots, up to 64 internal disks, support for external I/O, a new high performance interconnect capable of up to 304.2 GB/sec, and support for up to 24 domains.
7 Architecting Cost-Effective, Scalable Oracle Data Warehouses Sun SPARC Enterprise M9000 server 48 CPUs 768 GB RAM 96x HBA ports SAN Two Sun StorageTek 9990V systems 1792 disks 448x 4 Gb/sec. FC Figure 4. Large data warehouse up to 143.36 TB The Sun StorageTek 9990V system offers very high levels of enterprise functionality, performance, scalability, connectivity, and availability. It delivers controller-based storage virtualization of up to 32 PB of internal and external storage, tiered storage management for data lifecycle management, and up to 32 virtual partitions for optimization of cache, disk, and I/O. Performance scales up to 2.5 million I/Os and 81 GB/sec aggregate internal bandwidth, and supports remote replication for multidatacenter disaster recovery.. Partial reference configuration list Table 1 contains a partial listing of Sun s reference configurations. For a full list, contact a Sun sales representative. Table 1. Partial reference configuration list Size Server Storage Small 5.76 TB Small 17.92 TB Small 23.04 TB Medium 25.6 TB Medium 35.84 TB 1x Sun Fire X4450 server, 2 CPUs, 32 GB RAM, 4x HBA ports 1x Sun SPARC Enterprise T5440 server, 4 CPUs, 128 GB RAM, 16x HBA ports 4x Sun Fire X4450 servers, 2 CPUs, 32 GB RAM, 4x HBA ports each 1x Sun Fire X4600 M2 server, 8 CPUs, 128 GB RAM, 16x HBAs ports 2x Sun SPARC Enterprise T5440 servers, 4 CPUs, 128 GB RAM, 16x HBA ports each 2x Sun StorageTek 2540 arrays, 72 disks, 8x 4 Gb/sec host 1x Sun StorageTek 6540 array, 224 disks, 16x 4 Gb/sec host 8x Sun StorageTek 2540 arrays, 288 disks, 64x 4 Gb/sec host 2x Sun StorageTek 6540 arrays, 320 disks, 32x 4 Gb/sec host 2x Sun StorageTek 6540 array, 448 disks, 32x 4 Gb/sec host
8 Architecting Cost-Effective, Scalable Oracle Data Warehouses Size Server Storage Medium 35.84 TB Medium 53.76 TB Medium 71.68 TB Large 107.54 TB Large 143.36 TB Extra Large 184.32 TB Extra Large 215.04 TB 3x Sun SPARC Enterprise M5000 servers, 8 CPUs, 64 GB RAM, 8x HBA ports each 3x Sun SPARC Enterprise T5440 servers, 4 CPUs, 128 GB RAM, 16x HBA ports each 6x Sun SPARC Enterprise M5000 servers, 8 CPUs, 64 GB RAM, 8x HBA ports each 2x Sun SPARC Enterprise M8000 servers, 16 CPUs, 256 GB RAM, 32x HBA ports each 1x Sun SPARC Enterprise M9000-64 server, 48 CPUs, 768 GB RAM, 96x HBA ports 2x Sun SPARC Enterprise M9000 servers, 36 CPUs, 512 GB RAM, 64x HBA ports each 4x Sun SPARC Enterprise M8000 servers, 16 CPUs, 256 GB RAM, 32x HBA ports each 2x Sun StorageTek 6540 arrays, 448 disks, 32x 4 Gb/sec host 3x Sun StorageTek 6540 array, 672 disks, 48x 4 Gb/sec host 4x Sun StorageTek 6540 arrays, 896 disks, 64x 4 Gb/sec host interface 6x Sun StorageTek 6540 arrays, 1344 disks, 96x 4 Gb/sec host 2x Sun StorageTek 9990V systems, 1792 disks, 448x 4 Gb/sec host 2x Sun StorageTek 9990V systems, 2304 disks, 448x 4 Gb/sec host 12x Sun StorageTek 6540 arrays, 2688 disks, 192x 4 Gb/sec host Reliability and high availability To make data warehouses more available and reliable, Sun systems are equipped with redundant, hot-swappable components. The Solaris 10 OS includes other technologies such as self-healing, automatic reconfiguration around failed components, automatic restart of services, multipathing to disks and networks, and failover mechanisms. Solaris Predictive Self Healing the Solaris 10 OS can detect failing system components including CPUs, memory, and I/O and take them offline without requiring the database to go offline. Multipathing to disks Implementing higher levels of reliability and availability requires redundant host connectivity to storage devices. The Solaris 10 OS manages the failure of storage paths while maintaining host I/O connectivity through available secondary paths. IP Multipathing (IPMP) A feature in the Solaris 10 OS that enables IP fail-over and IP link aggregation. It helps manage network workloads and failures on the Sun servers. Service Management Facility A Solaris 10 OS feature that provides a mechanism for failed services to restart automatically in dependency order.
9 Architecting Cost-Effective, Scalable Oracle Data Warehouses Solaris Cluster software augments Oracle s already strong availability features by helping to ensure the least possible downtime, integrity of cluster node membership, and data integrity in the event of a failure. The Oracle RAC agent allows for maximum Oracle RAC uptime by reducing the time involved in detecting and recovering from a failure. Oracle Optimized Warehouse for Sun As part of the Oracle Optimized Warehouse Initiative, in addition to the reference configurations, Sun and Oracle have developed the Oracle Optimized Warehouse for Sun. Oracle Optimized Warehouses for Sun are fully configured data warehouse products that are sold and supported as a single product, thus reducing risk and deployment times. Oracle software is pre-installed, pre-configured, and tested on Sun and Oracle co-developed reference configurations based on Sun servers, storage, and the Solaris 10 OS. The pre-configured, calibrated, and validated solutions undergo a rigorous set of tests to produce balanced configurations that provide exceptional and predictable performance.
10 Architecting Cost-Effective, Scalable Oracle Data Warehouses Chapter 3 Summary Sun and Oracle have been working together for more than 25 years to deliver the ultimate in secure, reliable, and scalable enterprise-class data centers to thousands of customers around the world. With a shared commitment to open, standardsbased computing and market-leading innovations, such as grid computing, eco computing, and Java technology, Sun is a market share leader for Oracle Database and Applications. Together, Oracle s powerful data warehouses on Sun s highly scalable platforms, along with reference configurations for data warehouses, Oracle Optimized Warehouse for Sun, industry expertise, and consulting services can help enterprises quickly implement and deliver high performance, low cost, real-time data warehouse systems to create a competitive advantage. Getting started To start, use the reference configurations to get an idea of the systems and storage required for the amount of data and workload to be supported in the data warehouse. To integrate the solution into an entire BIDW solution, Sun can help with Sun Client Solutions for BIDW and Sun SM Solution Centers. Sun realizes that architecting an end-to-end infrastructure is quite a challenge and believes the people who intimately understand the technology are the best people to determine specific needs and design the right solution. Toward that end, the consultants in Sun Client Solutions for BIDW provide a business-focused, architecture driven approach that uniquely satisfies BIDW deployment challenges. Sun s BIDW solutions include requirements gathering, architecture design, migration, and performance tuning, along with a wide variety of applications for the Solaris OS. Sun Solution Centers can help take the guesswork out of implementing BIDW solutions. IT staff can try out concepts, size systems, test performance and scalability, and see working prototypes (proof-of-concept) on Sun s equipment all at no charge. They can leverage the combined expertise of Sun BIDW specialists and third-party vendors. Table 1. Web sites for more information. Description Sun and Oracle Oracle Optimized Reference Configurations for Sun and Oracle Optimized Warehouse for Sun Sun Solution Centers Sun Trade-In Program URL http://sun.com/oracle http://sun.com/oracle/owi http://sun.com/third-party/global/oracle/ competency/index.jsp http://www. sun.com/tradeins
Architecting Cost-Effective, Scalable Oracle Data Warehouses 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com 2009 All rights reserved. Sun, Sun Microsystems, the Sun logo, Java, Solaris, Sun Fire, Sun SPARC Enterprise, and Sun StorageTek are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Intel Xeon is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. AMD Opteron is a trademark or registered trademark of Advanced Micro Devices. Information subject to change without notice. Printed in USA SunWIN # 562758 05/09