An Oracle Technical White Paper October 2011 Oracle Optimized Solution for Oracle Secure Backup
Introduction... 1 Oracle Optimized Solution for Oracle Secure Backup... 2 Architecture Overview... 3 Reference Architecture Components... 4 Hardware Components... 6 Oracle Secure Backup Software... 6 StorageTek Tape Library and Tape Drive Products... 8 StorageTek ACSLS Manager Software... 11 Oracle Key Manager... 12 Implementation and Configuration... 13 Oracle Secure Backup Administrative Servers... 13 Oracle Secure Backup Administrative Servers with High Availability13 Oracle Secure Backup Media Server... 14 StorageTek Tape Drive and Library Products... 14 Storage Area Network... 15 Oracle Secure Backup Clients... 15 Software Components... 15 System Configuration and Tuning... 17 Performance Characterization... 19 Administrative Server Configurations... 20 Media Server Configurations... 20 Server I/O Performance... 21 Tape Drive Performance... 22 Backup Best Practices... 22 Oracle Exadata Backup Best Practices... 22 Sun ZFS Storage Appliance Backup Best Practices... 23 Sizing and Recommended Solutions... 23 Oracle Exadata... 23 Oracle SPARC SuperCluster... 26 Oracle Optimized Solutions That Rely Heavily on the Oracle Database 27
Oracle Optimized Solution for Oracle WebLogic Suite... 28 Oracle Optimized Solution for Oracle WebCenter Suite... 29 Conclusion... 31 About the Author... 31 References... 32 Appendix A Server Components in the Solution... 33 SPARC T-Series Servers... 33 Sun x86 Systems... 34
Introduction Virtually every company, regardless of size or industry, needs data archive capabilities. Datacenter administrators across all industry segments face similar archive challenges: explosive data growth, pressure to meet promised service levels and backup windows for users, forever archive data retention and retrieval requirements, and rising energy costs. Their data protection environments are becoming larger, more complex, and increasingly diverse. Administrators face the challenges of meeting the growing data backup requirements and providing a comprehensive backup solution, while maintaining costs and building a platform that will continue to meet future demands. This paper describes the Oracle Optimized Solution for Oracle Secure Backup. The solution includes optimized architectures for deploying Oracle Secure Backup to provide data protection for Oracle engineered systems, including Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solutions. These pretested, high-performance architectures can be used to accelerate data protection processing and management with breakthrough cost structures using the Oracle Secure Backup software powered by Oracle's servers, Oracle s operating systems, and Oracle s tape storage. These solutions deliver next-generation data protection for large and small environments with nearly no-single-point-of-failure high availability and the security of encryption. The flexible, multitier architecture provides virtually unlimited scalability with central management and end-to-end data protection across heterogeneous technologies. Architecture Overview on page 3 presents the hardware and software components used to implement the solution. Implementation and Configuration on page 13 describes the hardware configuration details and suggested tuning for the Oracle Solaris Operating System and Oracle Secure Backup software. Performance Characterization on page 19 includes details on the testing performed to help guide sizing and implementation recommendations. 1
Oracle Optimized Solution for Oracle Secure Backup The Oracle Optimized Solution for Oracle Secure Backup was designed as a solution for performing network backups of heterogeneous clients, including Oracle s engineered systems for Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solutions. The Oracle Secure Backup solution provides pretested, recommended configurations for a complete backup and recovery solution for different RTOs and different backup client types, as shown in Figure 1. This solution can provide complete end-to-end data protection for network backup clients or it can complement Oracle Recovery Manager (Oracle RMAN) and Oracle Active Data Guard data protection solutions using Oracle s Sun ZFS Storage Appliances with cost effective, long-term storage on high-performing encrypted tape. Figure 1. Providing the complete solution for any recovery time objective. The Oracle Optimized Solution for Oracle Secure Backup is a unique offering in that it is both low cost and high performance, making it a clear industry leader in terms of price/performance for backup and recovery. It is lower cost than any high-performance backup solution because software licensing costs for Oracle Secure Backup software are up to 75% less than the costs for most competitor products. Secondly, the solution enables the use of tape media, which is the lowest-cost storage media, and tape is an ideal media for storing older backup copies. The solution s performance scales nearly linearly using a building block approach with pretested server and storage configurations. This architecture also simplifies the solution implementation and makes it easier to do future upgrades to accommodate data growth. When the solution is deployed to backup-replicated Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solutions, it dramatically simplifies backup and recovery processing and management. These systems are typically deployed in replicated configurations, obviating the need to deploy complex third party de-duplication technologies. Complete data protection can be achieved through simple tape backups at both sites. 2
Achieving a compelling price/performance metric for this solution was a key objective. Another key objective was determining the performance and sizing characteristics of the proposed architectures. This evaluation included establishing the number of backup targets from Oracle's backup target portfolio that could be supported, along with an understanding of the configuration. Architecture Overview This solution features a scalable, multitier architecture including backup clients, media servers, administrative servers, and encrypted tape devices, as shown in Figure 2. Oracle Secure Backup software organizes the hosts and tape devices into three primary categories: An administrative domain, representing the network of hosts containing the data that is to be backed up and the backup and recovery management servers (Client and Administration Tiers in Figure 2) Hosts with attached tape devices on which backups are stored (Data Management Tier in Figure 2) Tape devices that are attached to the hosts (Storage Tier in Figure 2) Figure 2. Functional components of the Oracle Optimized Solution for Oracle Secure Backup. 3
Oracle Secure Backup software is used to provide encrypted tape-based data protection, archival, and recovery management for networked heterogeneous clients. Powered by either the Sun Fire X4270 M2, Sun Fire X44704470 M2, SPARC T3-1, or SPARC T3-2 servers along with Oracle tape archive devices, the solution scales to support an enterprise network with a large number of heterogonous networked clients, high-performing NAS devices, or high-performing appliance solutions such as Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solutions. This solution features Sun Fire X4270 M2/X4470 M2 or SPARC T3-1/T3-2 servers to create the optimal backup and recovery solution that best fits an organization s datacenter requirements and standards. Both platforms deliver best-in-class network backup performance and best-in-class reliability. The SPARC T-Series servers from Oracle can deliver up to 20% better network performance than the Sun Fire X4270 M2 and Sun Fire X4470 M2 server platforms. However, both types of platforms have similar bandwidth offerings in terms of the number PCI-e slots. To ease interoperability challenges and deliver more stability, it is recommended that the same platform found in the high-performing appliance solutions, such as Oracle Exadata, Oracle SPARC SuperCluster, and Optimized Solutions, be deployed when possible. All the recommended components in the solution feature nearly no-single-point-of-failure configurations in all areas of the environment, including servers, SAN, Ethernet and InfiniBand networks, tape library management, tape encryption management, and tape drives. Reference Architecture Components The primary logical components of Oracle Optimized Solution for Oracle Secure Backup include: Oracle Secure Backup administrative server Each administrative domain must have exactly one administrative server instance. During post-installation configuration, the administrative server must be configured with complete data regarding the other hosts in the administrative domain, their roles, and their attached tape devices. This configuration information is maintained in a set of configuration files stored on the administrative server. The administrative server runs the scheduler, which starts and monitors each backup job. The scheduler also keeps a backup catalog with metadata for all backup and restore operations performed in the administrative domain. Sun Fire X4270 M2 and SPARC T3-1 servers are recommended in this solution as administrative servers because they deliver outstanding administrative server performance in the smallest possible configuration of the server, providing considerable room for growth. It is recommended that administrative server and media server roles not be mixed. Oracle Secure Backup media server A media server is a host with at least one tape drive or library attached to it. It transfers data to or from a volume loaded on one of these tape devices. Single media servers can have attachments to multiple tape libraries and multiple media servers can share attachments to multiple tape libraries. 4
The media server performs the backup operations under the direction of the administrative server, with all backup data stored on its connected storage devices. Sun Fire X4270 M2 and SPARC T3-1 servers are recommended in this solution as media servers because they each offer the most I/O bandwidth for their platform type, a feature that dramatically improves media server performance. Both configurations also provide a great deal of headroom for growth in most environments. Larger environments in which a large amount of tape cloning is required may benefit from using Sun Fire X4470 M2 and SPARC T3-2 servers. However, these servers are too large for most environments. It is recommended that administrative server and media server roles not be mixed within the same server. Tape drives, tape encryption, and tape libraries A wide variety of tape drives and tape libraries are supported by the Oracle Secure Backup software. This solution features Oracle's StorageTek SL500, StorageTek SL3000, and StorageTek SL8500 modular library systems with Oracle's StorageTek T10000C and StorageTek LTO5 tape drives, where applicable. All the recommended architectures include the StorageTek SL3000 tape library because this library offers the lowest-cost configuration with no single point of failure. This solution features Oracle Key Manager for encryption key management for encrypted tape backups. Oracle Key Manager 2.3 makes it easy to implement and scale storage-based encryption for operational and archive data without unnecessary cost and complexity. This solution features Oracle s StorageTek Automated Cartridge System Library Software high-availability () solution for non-stop tape library management. Backup clients This solution focuses on backup and recovery for Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solutions, but it can also be used for network backups of any client platform that Oracle Secure Backup supports. Network switches Ethernet switches provide connections and link aggregation between the backup clients, media servers, and administrative servers. Network speeds of both 1 Gb/sec and 10 Gb/sec are supported. A private 10 GbE network or connections into the InfiniBand fabric of the solution is used to provide connections from the media servers to the network clients. SAN switches SAN switches are used to provide Fiber Channel connections between the Oracle Secure Backup media servers and the Fiber Channel archive devices. Although this solution includes multiple Oracle Secure Backup administrative and media servers, single media servers can be deployed to support different network backup workloads. This design provides a highly scalable solution, allowing additional media servers and additional archive devices to be added as needed to address scalability requirements. 5
Hardware Components All the recommended architectures in the solution feature common hardware components, as shown in Table 1. TABLE 1. ORACLE OPTIMIZED SOLUTION FOR ORACLE SECURE BACKUP COMMON HARDWARE COMPONENTS COMPONENT DESCRIPTION Administrative Servers Either Oracle s Sun x86 systems or SPARC T-Series servers Media Servers Either Sun x86 systems or SPARC T-Series servers Storage Area Network Brocade switches Oracle s Sun SPARC Enterprise M3000 servers and Sun Storage 6180 array Oracle Key Manager Oracle Key Manager appliances Tape Libraries StorageTek SL500, SL3000, and SL8500 tape libraries Tape Drives StorageTek T10000C or LTO5 tape drives The following sections describe the key components of the Oracle Optimized Solution for Oracle Secure Backup. Software components and tape hardware components are described along with their interaction with other components in the solution. Hardware server components are described in Appendix A. Oracle Secure Backup Software Oracle Secure Backup enables reliable data protection with file-system, NDMP, or Oracle RMAN backup to tape. It uses standard tape formats and supports every major tape drive and tape library across SAN, Ethernet networks, Oracle InfiniBand networks, and SCSI environments. Using Oracle Secure Backup on the network enables administrators to take data from a networked host running Oracle Secure Backup or a NAS device that supports NDMP, and back up that data on a tape device on the network. That data can include ordinary file-system files or databases backed up with Oracle RMAN. As part of the Oracle Optimized Solution for Oracle Secure Backup, Oracle Secure Backup provides scalable distributed backup and recovery capabilities. It reduces the complexity of a backup solution by: Integrating with the Oracle stack for maximum ease of use in a single Oracle solution to back up data from disk to tape Employing single-vendor technical support for database and file-system backup and recovery to tape Utilizing dynamic tape drive sharing for maximum tape drive utilization 6
Oracle Secure Backup eliminates integration challenges by providing ready-to-use tape management software that offers single-vendor support. Oracle Secure Backup also reduces costs. When using Oracle Secure Backup with Oracle RMAN to back up and recover databases and files to and from tape, no third-party tape management software is required. Oracle Secure Backup provides the media management layer needed to use tape storage with Oracle RMAN. Centralized administration, heterogeneous network support, and flexible scheduling simplify and automate protection of the entire Oracle environment, including database data and file-system data, such as the contents of the Oracle home directory. Oracle Solaris 10 Operating System The latest version of Oracle's enterprise-class operating system, Oracle Solaris 10, provides superior scalability, availability, manageability, and security. With its optimized network stack and support for advanced network protocols, Oracle Solaris 10 delivers high-performance networking. Innovative features new to Oracle Solaris 10, such as Oracle Solaris Predictive Self-Healing, support automatic diagnosis and recovery from hardware and application faults, maximizing system uptime, while the Oracle Solaris Dynamic Tracing Facility (DTrace) enables real-time application debugging and optimization. Oracle Solaris Containers With the advanced functionality of Oracle Solaris 10, multiple applications can be safely consolidated onto one system to increase utilization rates. Oracle Solaris 8, Oracle Solaris 9, and Oracle Solaris 10 applications can all be run in Oracle Solaris Containers (also known as Zones) on the same system, helping to save power, space, cooling, administrative, and support costs. Oracle Solaris Containers technology uses virtualization to allow organizations to share hardware resources while maintaining security isolation and other benefits of the one-application-per-server deployment model. Oracle Solaris Containers technology is part of a comprehensive offering of Oracle Solaris virtualization technologies. Oracle Solaris Cluster Oracle Solaris Cluster, built on the solid foundation of Oracle Solaris, provides high availability and disaster recovery solutions for local to global datacenters. It supports a large portfolio of mission-critical applications in physical and virtual environments. Oracle Solaris Cluster is integrated and thoroughly tested with Sun servers, storage, connectivity solutions, and Oracle Solaris 10 features. 7
Oracle Linux Oracle Linux combined with Oracle's Unbreakable Enterprise Kernel brings the latest Linux innovations to market, delivering extreme performance, advanced scalability, and reliability for enterprise applications. Optimized for Oracle, Oracle Linux is built and tested to run Oracle hardware, databases, and middleware. It is recommended for all enterprise applications and offers the following advantages: More than a 75 percent performance gain in OLTP performance tests over a Red Hat 5 Compatible Kernel 200 percent speedup of InfiniBand messaging 137 percent faster solid-state drive (SSD) access Optimized for large servers Improved power management and energy efficiency Fine-grained CPU and memory resource control Derived from the stable 2.6.32 mainline Linux kernel StorageTek Tape Library and Tape Drive Products The Oracle Optimized Solution for Oracle Secure Backup features the StorageTek SL500, StorageTek SL3000, and SL8500 modular library systems. These reliable modular library systems scale to suit a wide range of storage capacity requirements. The solution also features the StorageTek T10000C and StorageTek LTO5 tape drives. All of these components are described below. StorageTek SL500 tape library The StorageTek SL500 tape library scales from 30 to 575 LTO slots (or 24 to 494 mixed media slots) with a maximum capacity of more than 460 terabytes of uncompressed data in a standard 19-inch rackmounted cabinet. The library also supports up to 18 tape drives with a native throughput rate of more than 7 terabytes per hour. This library s capacity and scalability are often well suited for medium-sized business environments. 8
StorageTek SL3000 modular library system The StorageTek SL3000 modular library system, shown in Figure 3, offers an innovative, eco-efficient approach to midrange storage, providing more choice and control for rapidly changing environments. The StorageTek SL3000 modular library system scales from 200 to 5,925 1 cartridge slots and from 1 to 56 tape drives in a footprint that grows linearly in a rack environment. The RealTime Growth capability enables physical capacity to be installed in advance and tapped into incrementally with Capacity on Demand license keys. The StorageTek SL3000 modular library system delivers a native throughput rate of 48.4 TB/hour for maximum configurations. StorageTek SL8500 modular library system The StorageTek SL8500 modular library system, shown in Figure 3, can be shared across supercomputer, mainframe, Oracle Solaris, AS/400, Windows, Linux, and a variety of UNIX environments. Its "Any Cartridge Any Slot" technology enables any combination of supported drives and media. The choices include the enterprise StorageTek T10000, StorageTek T9840, and StorageTek T9940 tape drives, as well as LTO, SDLT, and DLT-S4 tape drives. The StorageTek SL8500 has nearly limitless scalability can currently scale to 100,000 slots (500 PB native storage) and up to 640 tape drives. Many libraries can be consolidated into one StorageTek SL8500 system, with modular and scalable capacity to meet rapid growth requirements. The StorageTek SL8500 modular library system delivers a native throughput rate of 552.9 TB/hour for maximum configurations. 1 Although the StorageTek SL3000 modular library system supports 5,925 cartridge slots, it is recommended to move to a StorageTek SL8500 modular library system if capacity needs are over 3000 slots. 9
Oracle Optimized Solution for Oracle Secure Backup Figure 3. StorageTek modular library systems. StorageTek Tape Drives The Oracle Optimized Solution for Oracle Secure Backup features StorageTek T10000C and LTO5 tape drives, shown in Figure 4. StorageTek T10000C Tape Drive StorageTek LTO5 Tape Drive Figure 4. StorageTek tape drives. StorageTek T10000C Tape Drive The StorageTek T10000C tape drive delivers a capacity of 5 TB uncompressed data and a throughput of 240 MB per second. These drive efficiencies help enable datacenters to store more data in the same footprint and shorten backup and recovery windows. The StorageTek T10000C tape drives support demanding, high-duty-cycle environments with dual-head, 32-channel technology, reducing the number of tape passes and extending head and media life. These tape drives connect using dual-port Fiber Channel options, for compatibility with most storage environments. 10
StorageTek LTO5 Tape Drive The StorageTek LTO5 tape drive provides fast data transfer rates and large capacity. The high-speed Fibre Channel interface further speeds data movement to shorten backup windows, reduce downtime, and speed recovery. LTO drives support two-generation backward compatibility, protecting technology investments. The open format supports a wide variety of media suppliers and automation systems. And the LTO Compliance Verification Entity, an independent company, verifies interchange across LTO vendors and product generations. StorageTek ACSLS Manager Software The Oracle Optimized Solution for Oracle Secure Backup features the solution. An example configuration is shown in Figure 5. Figure 5. example configuration. The solution leverages Oracle Solaris and Oracle Solaris Cluster high-availability software in a dual-server configuration. It houses the StorageTek ACSLS library management application on two separate servers. In the event of a hardware or software error, the solution automatically fails over to the redundant server, enabling continuous library control and access to data. is supported only by the Oracle Secure Backup software on media servers running Oracle Linux 5 32-bit kernels. 11
Oracle Key Manager The Oracle Optimized Solution for Oracle Secure Backup features an Oracle Key Manager solution that provides a highly available encryption key management system. An example configuration is shown in Figure 6. Figure 6. Oracle Key Manager Appliance example configuration. Oracle Key Manager 2.3 is designed with an emphasis on simplicity, security, and scalability to help organizations realize the following advantages: High Security. Oracle Key Manager is a hardened solution that provides FIPS 140-2 Level 3 compliance (with the Sun Crypto Accelerator 6000 PCIe Card) and secure key protection throughout key lifecycle, with a dedicated key management and delivery network. Interoperability. Open standards-based architecture supports diverse storage devices mainframe to open systems under a single storage key management system. High Availability. High availability is ensured by active n-node clustering, dynamic load balancing, and automated failover. Simplified Management. User-defined, policy-based automatic key management with secure client GUI makes it easy to administer the solution in one site or worldwide. Scalability. A single clustered Oracle Key Manager appliance pair can be used to manage thousands of storage devices and millions of encryption keys, making it easy to scale the solution without disruptions. 12
Implementation and Configuration While every deployment is unique, the configurations presented here provide a high-level overview of how the Oracle Optimized Solution for Oracle Secure Backup can be implemented. The following sections provide a high-level overview of the steps required to configure each component. Oracle Secure Backup Administrative Servers The SPARC T-Series and Sun x86 administrative server configuration requirements are generally the same. The general configuration requirements for Oracle servers used as the administrative servers are listed in Table 2. TABLE 2. ORACLE SECURE BACKUP ADMINISTRATIVE SERVER REQUIREMENTS COMPONENT CONFIGURATION REQUIREMENTS Operating System Install Oracle Solaris 10 (or optionally Oracle Linux 5.5 on x86 servers). Patch the operating system. Tune the operating system. Storage Configure storage for the Oracle Secure Backup installation and database. Oracle Secure Backup Administrative Servers with High Availability The SPARC T-Series and Sun x86 administrative server configuration requirements for high-availability are generally the same. The general configuration requirements for Oracle servers used as administrative servers with high availability are listed in Table 3. TABLE 3. ORACLE SECURE BACKUP ADMINISTRATIVE SERVER REQUIREMENTS FOR HIGH AVAILABILITY COMPONENT CONFIGURATION REQUIREMENTS Operating System Install Oracle Solaris 10 (or optionally Oracle Linux 5.5 on x86 servers). Patch the operating system. Tune the operating system. Storage Configure external cluster shared storage on the Sun Storage 6180 Array. Configure Oracle Solaris 10 Container Configure and install an Oracle Solaris Container on the shared cluster storage. Oracle Solaris Cluster Install and configure Oracle Solaris Cluster. Configure cluster resource group for Oracle Solaris Container HA management. 13
Oracle Secure Backup Media Server The SPARC T-Series and Sun x86 media server configuration requirements are generally the same. The general configuration requirements for Oracle servers used as media servers are listed in Table 4. TABLE 4. ORACLE SECURE BACKUP MEDIA SERVER REQUIREMENTS COMPONENT CONFIGURATION REQUIREMENTS Operating System Install Oracle Solaris 10 (or optionally Oracle Linux 5.5 on x86 servers). Patch the operating system. Tune the operating system. High-Performance Client Network Configure InfiniBand adapters or 10GbE adapters for client network access. Storage Area Network Configure the two switches with inter-switch links (ISLs) and switch configuration for HA. Configure zoning to connect the tape drives to the media servers. Storage Configure tape drives in the operating system. StorageTek Tape Drive and Library Products The StorageTek SL500, StorageTek SL3000, and StorageTek SL8500 modular library systems in the Oracle Optimized Solution for Oracle Secure Backup should be configured to be controlled by the library management servers. The general configuration requirements for StorageTek tape drive and library products are listed in Table 5. TABLE 5. STORAGETEK TAPE DRIVE AND LIBRARY PRODUCTS REQUIREMENTS COMPONENT CONFIGURATION REQUIREMENTS Library Network Configure dual network interfaces on supported libraries. Library Fiber Channel Configure dual Fiber Channel interfaces on supported libraries. Library Hardware Partitioning Configure hardware partitioning, if desired, on supported libraries. StorageTek ACSLS HA Configure tape drives in the operating system. Oracle Key Manager Configure Oracle Key Manager appliances for an HA configuration. Configure encryption network between the encryption appliances and tape drives. 14
Storage Area Network The Oracle Optimized Solution for Oracle Secure Backup features a Brocade Storage Area Network (SAN) using two 8 Gb switches. All Fiber Channel ports from the servers and storage devices, with the exception of the tape drives, should be connected to the switches using 8 Gb GBICs. The 4 Gb devices negotiate the port speed upon connection to 4 Gb. The media server Fiber Channel ports and tape drives are balanced across the two switches for performance and high availability. TABLE 6. STORAGE AREA NETWORK REQUIREMENTS COMPONENT CONFIGURATION REQUIREMENTS Switch Configure the switches with ISLs between the two switches. Fabric Configure zoning to logically connect the tape drives to the media servers. Oracle Secure Backup Clients Generally, there are no hardware configuration changes required on the client systems. Engineered systems may need additional cabling to connect to InfiniBand switches or 10 GbE ports depending on the configuration of the engineered system. Please see the "References" section for a list of links to best practices for client configurations with Oracle Secure Backup. Software Components The Oracle Secure Backup 10.3.3 software is used by all servers and clients in the backup environment. The Oracle Secure Backup administrative and media servers all run either Oracle Solaris 10 or Oracle Linux 5.5. Administrative servers configured for high availability and servers use Oracle Solaris Cluster 3.3. Additional software tuning was performed, as described in System Configuration and Tuning on page 17, to improve performance. Oracle Solaris 10 and Oracle Linux The Oracle Secure Backup administrative and media servers can run either Oracle Solaris 10 or Oracle Linux, with the latest recommended patches. For optimum performance, it is important to use the same version of the OpenFabrics Enterprise Distribution (OFED) InfiniBand drivers that are in use in the solution being attached to. Contact an Oracle representative to get up-to-date recommendations on all drivers and patches before implementing the solution. Oracle Secure Backup 10.3.3 Software The Oracle Secure Backup 10.3.3 or higher software should be installed on the administrative and media servers and on all clients in the network. 15
Server Firmware The firmware levels on the servers deployed as Oracle Secure Backup administrative and media servers should be verified and updated if necessary. The latest firmware revisions are available for download from http://support.oracle.com. Software Installation Overview A comprehensive installation procedure is outside the scope of this document. Briefly, the main steps include installing and configuring the Oracle Secure Backup 10.3.3 software on the various components in the backup network and performing system tuning to improve performance. 1. Install and configure the administrative server. a. Install Oracle Solaris 10 or Oracle Linux and all patches. b. Verify the server firmware is the most recent release, and update if needed. c. Install the Oracle Secure Backup 10.3.3 or higher software. 2. Install and configure the media server. a. Install Oracle Solaris 10 or Oracle Linux and all patches. b. Verify the server firmware is the most recent release, and update if needed. c. Install the Oracle Secure Backup 10.3.3 or higher software. d. Perform Oracle Solaris or Oracle Linux tuning and Oracle Secure Backup configuration and tuning, as described in "System Configuration and Tuning" on page 17. 3. Install and configure the clients. a. Install the Oracle Secure Backup 10.3.3 client software. b. Perform Oracle Secure Backup configuration, following Oracle Secure Backup best practices for the client type being backed up. See the "References" section for a list of links to best practices for client configurations with Oracle Secure Backup. 4. Configure Oracle Secure Backup policies and queues on the administrative server. See the Oracle Secure Backup 10.3.3 documentation set for detailed instructions on installing and configuring the software on the various systems in the backup network. 16
System Configuration and Tuning On Oracle Solaris 10 hosts, both the Oracle Solaris 10 operating system and the Oracle Secure Backup software are both configured for better backup performance. On Oracle Linux hosts, it is required only to configure Oracle Secure Backup for better backup performance. Oracle Solaris Operating System Configuration Various parameters defined within Oracle Solaris 10 are tuned on the Oracle Secure Backup media servers to achieve higher levels of performance. These settings should also be set on administrative servers if the servers are also acting as a media server. The following tuning parameters are based on Oracle Solaris 10 9/10. Current patches and/or later versions of Oracle Solaris 10 may require changes to these settings. Increasing the DMA buffers The ip_soft_rings parameters in the /etc/system file were modified from the default settings to the following: set ip:ip_squeue_fanout=1 set ip:ip_squeue_bind=0 set ip:ip_soft_rings_cnt=16 set ip:ip_soft_rings_10gig_vlan_cnt=16 set ip:ip_squeue_worker_wait=1 set ip:tcp_squeue_wput=1 set rpcmod:clnt_max_conns=8 This change increases the DMA buffers for the network interface cards, allowing the server to better utilize the available threads to handle interrupts. Two threads should be used for each 1 Gb port in use on the server, and eight threads should be used for each 10 Gb port in use on the server. The example shown above using a value of 16 could be used for eight 1 GbE ports or two 10 GbE ports. If the PCI Express dual 10 GbE card is used rather than the XAUI card in Oracle Solaris 10 5/09 and prior versions, the following parameter should also be modified: set ddi_msix_alloc_limit=8 NCA tuning and other system tuning for higher network performance In addition to setting the NCA parameters, some general system tuning can also benefit NCA performance. set sq_max_size=0 set ge:ge_intr_mode=1 set ddi_msix_alloc_limit=8 17
For more information on tuning Oracle Solaris NCA and general system tuning, please see the following resource: http://www.oracle.com/technetwork/serverstorage/solaris/documentation/index-jsp-135724.html#2. 10GbE adapter tuning (NXGE) The following settings should be used to tune the nxge driver in the /etc/system file. Note that using jumbo frames is recommended, but not required, to achieve the performance results found in this white paper. set nxge:nxge_msi_enable=2 set nxge:nxge_bcopy_thresh=1024 set nxge:nxge_jumbo_enable=1 NFS version 3 tuning If Sun ZFS Storage Appliances are used for basic disk pools, NFS version 3 should be tuned, as follows, in the /etc/system file: set nfs:nfs3_max_transfer_size=1048576 set nfs:nfs3_bsize=1048576 set nfs:nfs3_max_threads=16 The nfs:nfs3_max_transfer_size parameter sets the NFS version 3 transfer size to 1 M and the nfs:nfs3_bsize parameter sets the NFS version 3 buffer size to 1 M. The nfs:nfs3_max_threads parameter increases the number of threads NFS version 3 can use to process reads and writes from the shares on the Sun ZFS Storage Appliance, which increases overall NFS version 3 performance. Deferred ACK configuration Since backup servers receive large amounts of data and transmit network protocol acknowledgments (ACKs) most of the time, the Oracle Solaris kernel parameters were modified to reduce the number of outgoing ACKs for better performance. The following two ndd commands were executed on the Oracle Secure Backup media server: ndd -set /dev/tcp tcp_deferred_acks_max 16 ndd -set /dev/tcp tcp_local_dacks_max 16 18
The first command modifies the tcp_deferred_acks_max parameter, which specifies the maximum number of TCP segments received from remote destinations before an ACK is generated. This command dynamically changes the value for deferred ACKs from the default of two to the new value of 16. The second statement modifies the tcp_local_dacks_max parameter, and it dynamically changes the maximum number of deferred ACKs for the local subnet(s) from the default of eight to the new value of 16. TCP stack tuning The following ndd commands were executed on the Oracle Secure Backup servers to further tune the TCP stack for higher network performance: ndd -set /dev/tcp tcp_conn_req_max_q 16384 ndd -set /dev/tcp tcp_conn_req_max_q0 16384 ndd -set /dev/tcp tcp_max_buf 10485760 ndd -set /dev/tcp tcp_cwnd_max 10485760 ndd -set /dev/tcp tcp_recv_hiwat 400000 ndd -set /dev/tcp tcp_xmit_hiwat 400000 ndd -set /dev/nxge0 accept_jumbo 1 ndd -set /dev/nxge1 accept_jumbo 1 ndd -set /dev/nxge2 accept_jumbo 1 ndd -set /dev/nxge3 accept_jumbo 1 ndd /dev/nxge0 accept_jumbo ndd /dev/nxge1 accept_jumbo ndd /dev/nxge2 accept_jumbo ndd /dev/nxge3 accept_jumbo Note that using jumbo frames is recommended, but not required, to achieve the performance results found in this white paper. Performance Characterization Proof-of-concept testing was performed to determine the performance characteristics of the Oracle Secure Backup administrative and media servers. The testing was executed using Oracle Secure Backup 10.3.3 software to perform network-based backups from multiple test clients. The test plan called for several testing scenarios to verify the media server performance using tape storage devices sequentially. This approach was used to demonstrate that any of the Oracle server configurations featured in this solution are capable Oracle Secure Backup administrative servers and Oracle Secure Backup media servers. 19
Administrative Server Configurations The configurations for the Sun Fire X4270 M2 and SPARC T3-1 Oracle Secure Backup administrative servers are similar. In the test configuration, these servers included a single dual-port 8 Gb Fiber Channel HBA, which connected the Sun Storage 6180 Fibre Channel disk array for the Oracle Secure Backup database. They also included a single onboard 1 GbE port for network connectivity. Figure 7 shows the recommended PCI-e configuration for the SPARC T3-1 administrative servers. Figure 7. Recommended PCI-e configuration for the SPARC T3-1 administrative servers. Figure 8 shows the recommended PCI-e configuration for the Sun Fire X4270 M2 administrative servers. Figure 8. Recommended PCI-e configuration for the Sun Fire X4270 M2 administrative servers. Media Server Configurations The configurations for the Sun Fire X4270 M2 and SPARC T3-1 Oracle Secure Backup media servers are similar. These servers should be configured with 10 GbE or InfiniBand adapters depending on the solution to be backed up. It is important for the PCI-e adapters to be installed in to the slots as seen below. For example, if one network adapter and three Fibre Channel adapters are being installed, the network card would go into slot 0 and the three Fibre Channel adapters would go into slots 1, 2, and 4. Figure 9 shows the recommended PCI-e configuration for the SPARC T3-1 media servers. 20
Figure 9. Recommended PCI-e configuration for the SPARC T3-1 media servers. Figure 10 shows the recommended PCI-e configuration for the Sun Fire X4270 M2 media servers. Figure 10. Recommended PCI-e configuration for the Sun Fire X4270 M2 media servers. Server I/O Performance During testing of the Oracle Optimized Solution for Oracle Secure Backup, the recommended Oracle servers demonstrated outstanding network backup performance to StorageTek T10000C and StorageTek LTO5 tapes drives. Table 7 summarizes the performance of the servers in the solution. The maximum network backup performance listed in this table was achieved without using jumbo frames on the client network. TABLE 7. SUMMARY OF NETWORK BANDWIDTH MEDIA SERVER MAXIMUM NETWORK BACKUP Sun Fire X4270 M2 server Up to 3 GB/sec Sun Fire X4470 M2 server Up to 6 GB/sec SPARC T3-1 server Up to 3 GB/sec SPARC T3-2 server Up to 6 GB/sec 21
Tape Drive Performance Tape drive performance requires fast backup clients to achieve the native write performance of the tape drive. Even higher performance can be achieved with compressible data. All Oracle engineered systems, including Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solution environments perform well writing to tape. Depending on the data being backed up, the actual performance per tape drive could be higher than the native write rate of the tape drive. StorageTek T10000C and LTO5 Tape Drives Performance for the StorageTek T10000C and LTO5 tape drives is summarized in Table 8. Compressible data was not used for tape performance testing and is not shown. The performance shown in Table 8 could be increased by using compressible data. TABLE 8. STORAGETEK TAPE DRIVE PERFORMANCE TAPE DRIVE MODEL NATIVE WRITE RATE EXPECTED WRITE RATE IN THESE SOLUTIONS StorageTek LTO5 140 MB/sec 120 140+ MB/sec StorageTek T10000C 240 MB/sec 160 240+ MB/sec Backup Best Practices In-depth coverage of backup best practices for each solution is out of the scope of this document. The following sections show where to find backup best practices for each solution. Oracle Exadata Backup Best Practices All Oracle Exadata best practices are developed by the Oracle Maximum Availability Architecture (MAA) group. Oracle MAA is Oracle's best practices blueprint based on Oracle high availability technologies, extensive validation performed by the Oracle MAA development team, and the accumulated production experience of customers who have successfully deployed business-critical applications on Oracle solutions. MAA best practices span the Oracle Exadata Database Machine, Oracle Database, Oracle Fusion Middleware, Oracle Applications, Oracle Enterprise Manager Grid Control, and Oracle Partners. For more information on Oracle Exadata backup best practices, please see the following resource: http://www.oracle.com/technetwork/database/features/availability/maa-090890.html. 22
Sun ZFS Storage Appliance Backup Best Practices The recommended backup solution for some Oracle SPARC SuperCluster solutions, Oracle Optimized Solution for Oracle WebLogic Suite, and Oracle Optimized Solution for Oracle WebCenter Suite environments compliments the Sun ZFS Storage Appliance Solution for Database Backup and Recovery by adding tape storage and tape storage management. The capacity of onsite backup storage can thus be extended, or offsite copies can be created for long-term retention. It is recommended to backup the primary site data and the remote site data to tape using Oracle RMAN backups and the ZFS Storage Appliance s NDMP ZFS backup type feature through the Oracle Secure Backup media servers. For more information on Oracle Fusion Middleware backup best practices, please see the following resource: http://www.oracle.com/technetwork/database/features/availability/fusion-middlewaremaa-155387.html. Also see the following resource for Sun ZFS Storage Appliance Solution for Database Backup and Recovery: http://www.oracle.com/us/products/servers-storage/storage/unified-storage/ocom-sunzfs-storage-solution-bur-171686.html. Sizing and Recommended Solutions The following sections outline the recommend backup solutions for Oracle engineered systems, including Oracle Exadata, Oracle SPARC SuperCluster, and Oracle Optimized Solution environments. Most of the sizing is based on an eight-hour backup window, with one exception. Oracle Exadata Database Machines that are configured with high-capacity disks are based on a 16-hour backup window. Contact an Oracle representative to calculate solution sizing for different backup window lengths. The bandwidth quotes for each solution are based on the average range of tape drive performance for the recommended tape drives in this solution. The averages used are 240 MB/sec for StorageTek T10000C tape drives and 140 MB/sec for StorageTek LTO5 tape drives. Oracle Exadata In this solution, the database backups are performed with Oracle RMAN handing backup data to the Oracle Secure Backup software on the media server, which then writes the data to locally attached tape devices or remotely attached tape devices. 23
Figure 11. Oracle Exadata environments utilize Oracle RMAN and media servers to write to tape through Oracle Secure Backup. TABLE 9. ORACLE EXADATA WITH HIGH-PERFORMANCE DISK RECOMMENDED ARCHITECTURES X2-2 QUARTER RACK X2-2 HALF RACK X2-2 FULL RACK X2-8 FULL RACK ADMINISTRATIVE 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 GB SERVERS RAM MEDIA SERVERS 2 x Sun Fire X4270 M2 32 2 x Sun Fire X4270 M2 32 2 x Sun Fire X4270 M2 32 2 x Sun Fire X4270 M2 32 GB RAM TAPE DRIVES 3 X StorageTek T10000C 4 X StorageTek T10000C 8 X StorageTek T10000C 8 X StorageTek T10000C (preferred) (preferred) (preferred) (preferred) or- 5 X StorageTek LTO5 7 X StorageTek LTO5 13 X StorageTek LTO5 13 X StorageTek LTO5 TAPE LIBRARY ORACLE KEY 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager MANAGER Appliances Appliances Appliances Appliances MAXIMUM BACKUP BANDWIDTH AS CONFIGURED ABOVE: X2-2 QUARTER RACK X2-2 HALF RACK X2-2 FULL RACK X2-8 FULL RACK BACKUP 480 720 MB/sec 640 960 MB/sec 1280 1920 MB/sec 720 2160 GB/sec BANDWIDTH 1.6 2.5 TB/hr 2.2 3.3 TB/hr 4.4 6.6 TB/hr 4.4 6.6 TB/hr 24
TABLE 10. ORACLE EXADATA WITH HIGH-CAPACITY DISK RECOMMENDED ARCHITECTURES X2-2 QUARTER RACK X2-2 HALF RACK X2-2 FULL RACK X2-8 FULL RACK ADMINISTRATIVE 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 2 x Sun Fire X4270 M2 8 GB SERVERS RAM MEDIA SERVERS 2 x Sun Fire X4270 M2 32 2 x Sun Fire X4270 M2 32 3 x Sun Fire X4270 M2 32 3 x Sun Fire X4270 M2 32 GB RAM TAPE DRIVES 5 X StorageTek T10000C 13 X StorageTek 16 X StorageTek T10000C 16 X StorageTek T10000C (preferred) T10000C (preferred) (preferred) (preferred) 9 X StorageTek LTO5 21 X StorageTek LTO5 28 X StorageTek LTO5 28 X StorageTek LTO5 TAPE LIBRARY ORACLE KEY 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager MANAGER Appliances Appliances Appliances Appliances MAXIMUM BACKUP BANDWIDTH AS CONFIGURED ABOVE: X2-2 QUARTER RACK X2-2 HALF RACK X2-2 FULL RACK X2-8 FULL RACK BACKUP 800 1200 MB/sec 2080 3120 MB/sec 2560 3840 MB/sec 2560 3840 MB/sec BANDWIDTH 2.7 4.1 TB/sec 7.1 10.7 TB/hr 8.8 13.2 TB/hr 8.8 13.2 TB/hr The operating systems used on each server are identified in Table 11: TABLE 11. OPERATING SYSTEMS ON EACH SERVER IN THE SOLUTION ADMINISTRATIVE SERVERS Oracle Solaris 10 9/10 or higher MEDIA SERVERS Oracle Linux 5.5 or higher 25
Oracle SPARC SuperCluster In Oracle SPARC SuperCluster environments, database backups are performed by Oracle RMAN handing backup data to the Oracle Secure Backup software on the media server, which then writes the data to tape. Figure 12. Oracle SPARC SuperCluster environments utilize Oracle RMAN and media servers to write to tape through Oracle Secure Backup. TABLE 12. ORACLE SPARC SUPERCLUSTER RECOMMENDED ARCHITECTURES M-SERIES T-SERIES 8 FULL RACKS ADMINISTRATI VE SERVERS 2 x SPARC T3-1 8 2 x SPARC T3-1 8 2 x T3-1 SPARC 8 MEDIA SERVERS 2 x SPARC T3-1 32 2 x SPARC T3-1 32 16 x SPARC T3-1 32 TAPE DRIVES 15 X StorageTek T10000C (preferred) 15 X StorageTek T10000C 110 X StorageTek T10000C (preferred) (preferred) 25 X StorageTek LTO5 188 X StorageTek LTO5 25 X StorageTek LTO5 TAPE LIBRARY ORACLE KEY MANAGER 2 X Oracle Key Manager Appliances 2 X Oracle Key Manager Appliances 2 X Oracle Key Manager Appliances MAXIMUM BACKUP BANDWIDTH AS CONFIGURED ABOVE: M-SERIES T-SERIES 8 FULL RACKS 2 X MEDIA 2560 3840 MB/sec 2560 3840 MB/sec 18 28 GB/sec SERVERS 8.8 13.2 TB/hr 8.8 13.2 TB/hr 64.7 97 TB/hr 26
The operating systems used on each server are identified in Table 13: TABLE 13. OPERATING SYSTEMS ON EACH SERVER IN THE SOLUTION ADMINISTRATIVE SERVERS Oracle Solaris 10 9/10 or higher MEDIA SERVERS Oracle Linux 5.5 or higher Oracle Optimized Solutions That Rely Heavily on the Oracle Database The following Oracle Optimized Solutions rely heavily on Oracle Database and often contain a large database that must be backed up quickly and securely: Oracle Optimized Solution for Oracle Database Oracle Optimized Solution for Oracle E-Business Suite Oracle Optimized Solution for Siebel Customer Relationship Management (CRM) Oracle Optimized Solution for PeopleSoft Human Capital Management (HCM) For these solutions, the database backups are performed by Oracle RMAN handing backup data to the Oracle Secure Backup software on the server, which then writes the data to tape on the media server. Figure 13. Oracle Optimized Solution environments that rely heavily on the Oracle Database can utilize Oracle RMAN and media servers to write to tape through Oracle Secure Backup. TABLE 14. RECOMMENDED ARCHITECTURES FOR ORACLE OPTIMIZED SOLUTIONS THAT RELY HEAVILY ON THE ORACLE DATABASE SMALL MEDIUM LARGE X-LARGE ADMINISTRATI VE SERVERS 2 x SPARC T3-1 8 2 x SPARC T3-1 8 2 x SPARC T3-1 8 2 x SPARC T3-1 8 MEDIA SERVERS 2 x SPARC T3-1 32 2 x SPARC T3-1 32 2 x SPARC T3-1 32 2 x SPARC T3-1 32 TAPE DRIVES 2 X StorageTek T10000C 2 X StorageTek T10000C 4 X StorageTek T10000C 9 X StorageTek T10000C (preferred) (preferred) (preferred) (preferred) 27
TABLE 14. RECOMMENDED ARCHITECTURES FOR ORACLE OPTIMIZED SOLUTIONS THAT RELY HEAVILY ON THE ORACLE DATABASE SMALL MEDIUM LARGE X-LARGE 3 X StorageTek LTO5 3 X StorageTek LTO5 7 X StorageTek LTO5 15 X StorageTek LTO5 TAPE LIBRARY ORACLE KEY 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager 2 X Oracle Key Manager MANAGER Appliances Appliances Appliances Appliances MAXIMUM BACKUP BANDWIDTH AS CONFIGURED ABOVE: SMALL MEDIUM LARGE X-LARGE 2 X MEDIA 320 480 MB/sec 320 480 MB/sec 640 960 MB/sec 1440 2160 MB/sec SERVERS 1.1 1.6 TB/hr 1.1 1.6 TB/hr 2.2 3.3 TB/hr 4.9 7.4 TB/hr The operating systems used on each server are identified in Table 15. TABLE 15. OPERATING SYSTEMS ON EACH SERVER IN THE SOLUTION ADMINISTRATIVE SERVERS Oracle Solaris 10 9/10 or higher MEDIA SERVERS Oracle Linux 5.5 or higher Oracle Optimized Solution for Oracle WebLogic Suite In the Oracle Optimized Solution for Oracle WebLogic Suite, the storage backups are performed by backing up the Sun ZFS Storage Appliance in the solution with NDMP or file-system backups that write to tape through the Oracle Secure Backup media servers. File-system backups require the media servers to mount NFS shares on the Sun ZFS Storage Appliance. Figure 14. Oracle Optimized Solution for Oracle WebLogic Suite utilizes NDMP or file-system backups to write to tape. 28