An Oracle White Paper July Oracle Optimized Solution for Backup and Recovery

Transcription

1 An Oracle White Paper July 2013 Oracle Optimized Solution for Backup and Recovery

2 Introduction... 1 Oracle Optimized Solution for Backup and Recovery... 2 Addressing Data Recovery Needs and Requirements... 3 Architecture Overview... 4 Oracle Optimized Solution for Backup and Recovery Software Components... 4 Oracle Optimized Solution for Backup and Recovery Hardware Components... 8 Reference Architecture Components in Tape Environments Third-Party Software Support Oracle Secure Backup Backup Best Practices Best Practices for Accomplishing Recovery Time and Recovery Point Objectives Using a Combination of Software Tools Best Practices for Backup and Recovery in Virtualized Environments or with Unstructured Data Best Practices for Backup and Recovery in Virtualized Environments and Unstructured Data with NDMP Best Practices for Backup and Recovery of Operating Systems Oracle Solaris Backing Up Domains Backing Up Oracle Solaris Zones Best Practices for Database Backups (Oracle Database 11gR2 or Higher) Always Use a Fast Recovery Area Additional Database Configuration Best Practices Oracle RMAN Configuration Best Practices Restore and Recovery Best Practices Best Practices for Performing Local Disk-Based Backups with Oracle RAC... 22

3 Best Practices for Performing Local Disk-Based Backups with Oracle RAC One Node Database Best Practices for Disk-Based Backups Database Best Practices for Tape-Based Backups Best Practices for Tape-Based Backups Network Configuration Best Practices Back Up the Oracle Secure Backup Catalog Best Practices for Oracle Engineered Systems Backups SuperCluster Backup Best Practices Oracle Exadata Backup Best Practices Oracle Database Appliance Backup Best Practices Best Practices for the Oracle Sun ZFS Backup Appliance Database Backup and Restore Key Best Practices with a Sun ZFS Storage Appliance and Oracle Engineered Systems Configure Network Connectivity Follow Recommended Best Practices for Database Backup and Restore Tune the Oracle Database Instance for Oracle RMAN Configure Oracle RMAN Best Practices Using an External Sun ZFS Storage Appliance or Oracle Exadata Storage Expansion Rack Use ZFS Snapshots Using Non-Exadata Storage for Oracle SuperCluster and Oracle Exadata Database Machine Backups Providing Backup and Recovery Savings and High Availability with Remote DR Sites Conclusion References... 35

4 Introduction Oracle Engineered Systems and Oracle Optimized Solutions offer massive scalability and performance for today's larger and more complex workloads and increasingly diverse data environments. Engineered, optimized, and integrated, these systems provide the radical processing speeds, significantly faster deployments, and simplified data center operations needed to achieve unmatched enterprise performance levels. Combining best-of-breed hardware and software components with game-changing technical innovations, Oracle Engineered Systems and Oracle Optimized Solutions deliver such massive capacity and extreme performance that they require a new way of thinking about backup and recovery. With ownership of both the hardware and software technology, Oracle is in the unique position of being able to plan, design, and execute backup solutions for these powerful and scalable systems. This paper describes the Oracle Optimized Solution for Backup and Recovery, with particular focus on Oracle Engineered Systems, including Oracle Exadata Database Machine (Oracle Exadata), Oracle Database Appliance, Oracle SuperCluster, and Oracle Optimized Solutions. This pretested, high-performance backup solution can be used to accelerate data protection processing and management with breakthrough cost structures using Oracle software powered by Oracle's servers, operating systems, and tape storage. Delivering next-generation data protection with redundancy, high availability, and the security of encryption, the Oracle Optimized Solution for Backup and Recovery offers a flexible, multitier architecture for large and small environments. The solution also provides virtually unlimited scalability with centralized management and end-to-end data protection across heterogeneous technologies. 1

5 Oracle Optimized Solution for Backup and Recovery The Oracle Optimized Solution for Backup and Recovery is designed as a complete solution for performing network backups of heterogeneous clients. These clients include both Oracle Engineered Systems and Oracle Optimized Solutions. Leveraging built-in Oracle integration, the Oracle Optimized Solution for Backup and Recovery provides pretested, recommended configurations for complete backup and recovery solutions. These solutions address different recovery time objectives and different backup client types, as shown in Figure 1. The architecture is designed to simplify solution implementation and facilitate future upgrades as needed for data growth. With complete end-to-end data protection for network backup clients, the Oracle Optimized Solution for Backup and Recovery can also complement Oracle Recovery Manager (Oracle RMAN) and Oracle Active Data Guard data protection solutions using Oracle s Sun ZFS Storage Appliances for cost effective, long-term storage on high-performing encrypted tape. The Oracle Optimized Solution for Backup and Recovery is a comprehensive solution for Oracle Database backups with support for Oracle Database 9i or higher. Figure 1. The Oracle Optimized Solution for Backup and Recovery supports different backup client types. The Oracle Optimized Solution for Backup and Recovery is a unique offering in that it is both low cost and high performance, making it a clear industry leader in terms of price/performance for backup and recovery. Indeed, Oracle Secure Backup software licensing costs can be significantly less than the costs of comparable competitive products, because the tape management software component carries a 2

6 low-cost, one-time software licensing fee per tape drive used. Using solutions that do not rely on tape backups can save on the costs of licensing tape management software as well. For example, backups that utilize Sun ZFS Storage Appliances or Oracle Exadata storage systems are disk-only solutions and do not incur software licensing costs. In addition, the solution enables the use of tape the lowest-cost storage media, which is ideal for storing older backup copies. Capable of scaling nearly linearly, the solution uses a building block approach that consists of pretested server and storage configurations. When the solution is deployed to backup Oracle Engineered Systems and Oracle Optimized Solutions, it dramatically simplifies backup and recovery processing and management. Typically deployed in replicated configurations, these solutions eliminate the need to deploy complex third party deduplication technologies as simple tape and disk backups can be performed at one or both sites for complete data protection. Addressing Data Recovery Needs and Requirements When preparing data center backup and other data protection strategies, special attention must be paid to the recovery point objectives and recovery time objectives for all data and applications. The recovery point objective defines the maximum acceptable level of data loss following an unplanned event natural or man-made that can cause data to be lost. Measured backwards in time, and starting from the instant the event occurs, the recovery point objective is specified in seconds, minutes, hours, or days. It is used to help administrators select optimal disaster recovery technologies and procedures, and also helps to determine the minimum frequency for performing backups. For example, Wall Street trading firms need a recovery point objective of seconds or less, and must provide for redundant, remote systems to ensure that transactions are not lost. Organizations with more static data will not have such granular recovery point objectives. The recovery time objective is a period of time within which IT capabilities must be restored following an unplanned event. IT departments determine recovery time objectives by calculating the amount of revenue lost per unit of time as a result of the event, or by determining the maximum amount of time that an organization can afford to be without its IT functionality. Figure 2 illustrates various data center backup attributes, various Oracle backup and storage hardware, and which hardware addresses the various attributes. These factors should be kept in mind when evaluating backup solutions for Oracle Engineered Systems and Oracle Optimized Solutions. 3

7 Figure 2. The Oracle Optimized Solution for Backup and Recovery addresses a range of retention needs as well as requirements for recovery time (RTO) and recovery point objectives (RPO). Architecture Overview The Oracle Optimized Solution for Backup and Recovery features a scalable, multitier architecture that includes backup clients, media servers, administrative servers, Oracle disk devices, and encrypted tape devices. Designed to be software-agnostic, the Oracle Optimized Solution for Backup and Recovery can work with Oracle RMAN, Oracle Secure Backup, Symantec NetBackup, or other third-party backup software. For illustration purposes, this paper refers to use of Oracle Secure Backup software throughout. Note that in disk-only environments such as those using Oracle Exadata storage, Oracle Exadata Storage Expansion Racks, or Sun ZFS Storage Appliances no backup software is required. Disk backups can be completed using the operating system and Oracle Database tools such as Oracle RMAN alone. Oracle Optimized Solution for Backup and Recovery Software Components The following sections detail the software components of the Oracle Optimized Solution for Backup and Recovery. Oracle Solaris 11 Operating System Oracle Solaris 11 is the first operating system (OS) engineered for enterprise clouds. With built-in virtualization, Oracle Solaris 11 offers fast, intelligent provisioning capabilities for rapid service setup and maintenance. The operating system provides fully integrated security for users, applications, and devices with fine-grained delegated management and the latest security standards. Scalable data management is achieved with Oracle Solaris ZFS, the innovative default file system in Oracle Solaris 11. Oracle Solaris ZFS brings advanced storage features such as built-in de-duplication, 4

8 encryption, and thin provisioning to enterprise servers, and Oracle Solaris ZFS snapshots help to deliver boot environments that enable fail-safe updates and self-recovery. Oracle Solaris 11 offers mission-critical infrastructure that simplifies software lifecycle management and cloud-scale data management while providing advanced protection for public, private and hybrid cloud environments. New features increase performance, reliability, security, efficiency, and virtualization, as well as preserving full compatibility with a vast portfolio of existing third-party products as well as internally developed customer applications. Oracle Solaris 11 is engineered jointly with Oracle applications and middleware to deliver unique features that increase performance, streamline management, and automate support for Oracle deployments. Oracle Linux For those sites using Oracle x86 servers, Oracle Linux brings the latest Linux innovations to market, delivering extreme performance, advanced scalability, and reliability for enterprise applications and systems. Optimized for Oracle, Oracle Linux is built and tested to run Oracle hardware, databases, and middleware and is recommended for all enterprise applications. Oracle's StorageTek ACSLS Software Oracle's StorageTek Automated Cartridge System Library Software (StorageTek ACSLS ) solution provides a strategic, centralized library management solution. Figure 3 shows two environments one without StorageTek ACSLS and one after deploying StorageTek ACSLS. Figure 3. StorageTek ACSLS provides centralized management and high availability for StorageTek Tape Libraries. Without StorageTek ACSLS Manager software, organizations must grapple with distributed management, multiple administrators and libraries, excess environmental consumption, poor resource utilization, and higher total cost of ownership. Deploying StorageTek ACSLS Manager software as part of a tape management solution enables centralized management of a single library by a single administrator, with lower environmental consumption, standard policy management, balanced workload and resource utilization, maximum storage efficiency, and lower cost of ownership. In 5

9 addition, StorageTek ACSLS Manager software can help tape environments to meet high-availability objectives. Oracle Key Manager The Oracle Optimized Solution for Backup and Recovery features an Oracle Key Manager solution that provides a highly available encryption key management system for encrypted tape backups. Oracle Key Manager 2.5 makes it easy to implement and scale storage-based encryption for operational and archive data without unnecessary cost and complexity. A sample configuration is shown in Figure 4. Oracle Key Manager 2.5 is designed with an emphasis on simplicity, security, and scalability to help organizations realize the following advantages: High Security. A hardened solution, Oracle Key Manager provides FIPS Level 3 compliance (with the SCA 6000 Cryptographic Card) and secure key protection throughout key lifecycle, with a dedicated key management and delivery network. Interoperability. An open, standards-based architecture supports diverse storage devices from mainframes to open systems under a single storage key management system. High Availability. Active n-node clustering, dynamic load balancing, and automated failover ensure high availability. Simplified Management. A secure client GUI facilitates solution administration whether it is at one site or worldwide through user-defined, policy-based automatic key management. Scalability. A single clustered Oracle Key Manager appliance pair can be used to manage thousands of storage devices and millions of encryption keys, making the solution scale easily and non-disruptively. 6

10 Figure 4. An Oracle Key Manager Appliance sample configuration. Oracle's StorageTek Tape Analytics Software Oracle s StorageTek Tape Analytics software provides an intelligent monitoring application that proactively captures library, drive, and media health metrics and runs analytical calculations on these data elements. As a result, the software can eliminate library, drive, and media errors and empower tape storage administrators to make proactive decisions about tape environments prior to device failures. Figure 5 illustrates how the StorageTek Tape Analytics software collects data from libraries, drives, and media. Figure 5. StorageTek Tape Analytics software simplifies tape monitoring. 7

11 Oracle Optimized Solution for Backup and Recovery Hardware Components All the recommended architectures in the solution feature common hardware components, as shown in Table 2. TABLE 2. ORACLE OPTIMIZED SOLUTION FOR BACKUP AND RECOVERY COMMON HARDWARE COMPONENTS COMPONENT DESCRIPTION Storage Area Network Brocade switches Administrative Servers Either Oracle s Sun x86 systems or SPARC T-Series servers Media Servers Either Sun x86 systems or SPARC T-Series servers Tape Libraries Oracle;s StorageTek SL150, SL3000, and SL8500 tape libraries Tape Drives Oracle;s StorageTek T10000C or LTO5 or LTO6 tape drives StorageTek ACSLS A single Oracle server running the StorageTek ACSLS software Oracle Key Manager Oracle Key Manager appliances StorageTek Tape Analytics software A single Oracle server running the StorageTek Tape Analytics software Oracle's Sun ZFS Storage Appliance Backup and recovery solution for Oracle Engineered Systems in customizable small, medium, and large configurations Oracle Exadata Storage Expansion Rack Quarter, half, or full rack configurations that add storage capacity to an Oracle Exadata Database Machine or Oracle SuperCluster Network Infrastructure Ethernet switches provide connections and link aggregation between the backup clients, media servers, and administrative servers. Network speeds of both 1 Gigabit per second and 10 Gigabits per second are supported. Connections from the media servers to the network clients are provided through a private 10 Gigabit Ethernet network or connections into the solution's InfiniBand fabric. Built-in Oracle InfiniBand switches are used to provide connectivity within Oracle Engineered Systems. InfiniBand eliminates the physical complexity of multiple interconnects per system and provides immense bandwidth and high-speed connectivity to support high-speed transfers of massive amounts of data. With a high throughput, low latency, and scalable fabric suitable for fabric consolidation of interprocess communication, network, and storage, InfiniBand delivers up to 63 percent higher transactions per second for Oracle RAC over Gigabit Ethernet networks. SAN switches provide Fiber Channel connections between the Oracle Secure Backup media servers and the Fiber Channel archive devices. 8

12 Oracle's SPARC T-Series Servers The optimized solution features two members of Oracle's SPARC T-Series server family SPARC T4-1 and SPARC T5-2 servers (Figure 6). Powered by Oracle's SPARC T4 and T5 processors, these servers combine unmatched network performance and scalability with compact, energy-efficient form factors. Offering a five-fold improvement in single-thread performance over Oracle's previousgeneration SPARC T3 servers, SPARC T4 and SPARC T5 servers are ideal for large-scale applications, enterprise-wide consolidation, and database projects that require extreme reliability, availability, and security. Figure 6. SPARC T4-1 and SPARC T5-2 servers offer the highest levels of performance, scalability, and compute density in a versatile two-socket system. Oracle's Sun x86 Systems The solution also features three members of Oracle's Sun x86 system family Sun Server X3-2, Sun Server X3-2L, and Sun Server X2-4 (Figure 7) for small-, medium-, and large-sized environments. Oracle's comprehensive, open standards-based Sun x86 systems provide the best platform to run racle software and offer enhanced reliability for data center environments. Only Oracle provides an optimized hardware and software stack that comes complete with choice of OS, virtualization software, and cloud management tools all at no extra charge. Oracle's optimized hardware and software stack has enabled an enormous performance gain in its engineered systems and delivered world-record benchmarks results. Figure 7. Sun X3-2, X3-2L, and X2-4 servers meet requirements for small, medium, and large configurations. 9

13 Oracle Exadata Storage Expansion Rack The Oracle Exadata Storage Expansion Rack (see Figure 8) is engineered to be the simplest, fastest, and most robust way to add storage capacity to Oracle Exadata or Oracle SuperCluster. Available in quarter-, half-, or full-rack configurations, the Oracle Exadata Storage Expansion Rack can be tailored for specific backup requirements, and offers unique advantages when used as a backup destination. A full database backup of uncompressed data written to mirrored disk in an Oracle Exadata Storage Expansion Rack can be created at up to 27 TB per hour. Incremental database backups can achieve rates of hundreds of terabytes per hour, and incremental backups of Hybrid Columnar Compressed data can reach rates of petabytes per hour. Figure 8. The Oracle Exadata Storage Expansion Rack provides an optimal means for expanding capacity. Using an Oracle Exadata Storage Expansion Rack as a disk backup target can meet the most stringent requirements for recovery time objectives because the backup is usable directly without loss of performance and without having to perform a restore. This unique capability is only available when backing up to an Oracle Exadata Storage Expansion Rack. A strong solution for stringent recovery point objective requirements as well, the Oracle Exadata Storage Expansion Rack offers the smallest granularity of any of Oracle's backup solutions. Oracle's Sun ZFS Storage Appliance Oracle's Sun ZFS Storage Appliance (Figure 9) provides flexible, high-performance, and lowest cost backup and recovery solution for Oracle Engineered Systems. Offering unmatched Oracle integration, high performance, efficiency, simplified management, and low TCO, Sun ZFS Storage Appliances are available in customizable small, medium, and large configurations to suit varied backup requirements. 10

14 Figure 9. The lowest-cost backup and recovery solution for Oracle Engineered Systems is Oracle's Sun ZFS Storage Appliance. Oracle's Sun ZFS Backup Appliance Oracle's preconfigured, tested, and validated Sun ZFS Backup Appliance is a Sun ZFS Storage Appliance specifically designed and tuned for backup and recovery of Oracle Engineered Systems. When used with Oracle RMAN and other backup and recovery software, the Sun ZFS Backup Appliance safeguards against data corruption. The appliance can be directly connected to the Oracle Engineered Systems built-in InfiniBand fabric, delivering extremely fast backup and restore throughputs for narrow backup windows, and meeting recovery time objectives by providing timely recovery in the event of a disaster. Oracle's StorageTek Tape Library and Tape Drive Products Oracle tape storage solutions maximize the bandwidth of the Oracle Optimized Solution for Backup and Recovery. Providing fault isolation from production systems storage, the tape storage solutions enable easy movement of backup media off-site and provide cost-effective retention of multiple backup copies. Tape media can also be the most cost-effective method for an initial move of large amounts of data. In the event that a large database needs to be moved from site to site, such as to or from a secondary disaster recovery (DR) site, tapes can be shipped across the country quickly and cheaply, subsequently applying incremental backups to make the database current once again. The Oracle Optimized Solution for Backup and Recovery features Oracle's StorageTek SL150, StorageTek SL3000, and StorageTek SL8500 modular library systems. These reliable modular library systems scale to suit a wide range of storage capacity requirements. The solution also features Oracle's StorageTek T10000C and StorageTek LTO5 and LTO6 tape drives, where applicable. All the recommended architectures include the StorageTek SL3000 tape library because this library offers the lowest-cost configuration with no single point of failure. The tape libraries and tape drives are described below. 11

15 StorageTek SL150 modular tape library. The StorageTek SL150 tape library (Figure 10) scales from 30 to 300 LTO slots with a maximum capacity of more than 750 terabytes of uncompressed data (using LTO6 drives) in a standard 19-inch rackmounted cabinet. The library also supports up to 20 tape drives with a native throughput rate of more than 10 terabytes per hour. This library s capacity and scalability are designed for small and growing or mid-sized business environments. StorageTek SL3000 modular library system. The StorageTek SL3000 modular library system, shown in Figure 10, offers an innovative, eco-efficient approach to midrange storage, providing more choice and control for rapidly changing environments. The StorageTek SL3000 modular library system scales from 200 to 5,925 1 cartridge slots and from 1 to 56 tape drives in a footprint that provides linear growth in a rack environment. The StorageTek RealTime Growth capability enables physical capacity to be installed in advance and tapped into incrementally with Capacity on Demand license keys. The StorageTek SL3000 modular library system delivers a native throughput rate of 48.4 TB per hour for maximum configurations. StorageTek SL8500 modular library system. The StorageTek SL8500 modular library system, shown in Figure 10, can be shared across supercomputer, mainframe, Oracle Solaris, AS/400, Windows, Linux, and a variety of UNIX environments. Its Any Cartridge Any Slot technology enables any combination of supported drives and media. The choices include the enterprise StorageTek T10000 and StorageTek T9840 tape drives, as well as LTO and SDLT tape drives. The StorageTek SL8500 modular library system has nearly limitless scalability, and can currently scale to 100,880 slots (554.8 PB native storage) and up to 640 tape drives. Many libraries can be consolidated onto a single StorageTek SL8500 modular library system, with modular and scalable capacity to meet rapid growth requirements. The StorageTek SL8500 modular library system delivers a native throughput rate of TB per hour for maximum configurations. 1 Although the StorageTek SL3000 modular library system supports 5,925 cartridge slots, Oracle recommends moving to a StorageTek SL8500 modular library system if capacity needs are over 3000 slots. 12

16 Figure 10. Oracle's StorageTek modular library systems come in different sizes to accommodate data center needs. Oracle's StorageTek Tape Drives The Oracle Optimized Solution for Backup and Recovery features Oracle's StorageTek T10000C and LTO5 or LTO6 tape drives, shown in Figure 11. Figure 11. StorageTek tape drives. StorageTek T10000C Tape Drive. The StorageTek T10000C tape drive delivers a capacity of 5 TB uncompressed data and a throughput of 240 MB per second. These drive efficiencies help enable data centers to store more data in the same footprint and shorten backup and recovery windows. The StorageTek T10000C tape drives support demanding, high-duty-cycle environments with dualhead, 32-channel technology, reducing the number of tape passes and extending head and media life. These tape drives connect using dual-port Fiber Channel options for compatibility with most storage environments. StorageTek LTO5 Tape Drive. The StorageTek LTO5 tape drive provides fast data transfer rates with a throughput of 140 MB per second, and capacity of 1 TB. The high-speed Fibre Channel (FC) interface further speeds data movement to shorten backup windows, reduce downtime, and speed recovery. StorageTek LTO drives support two-generation backwards compatibility, protecting technology investments. The open format supports a wide variety of media suppliers and automation systems, and the LTO Compliance Verification Entity, an independent company, verifies interchange across LTO vendors and product generations. 13

17 StorageTek LTO6 Tape Drive. The StorageTek LTO6 tape drive is Oracle's latest addition to its portfolio of tape drives. With a capacity of 2.5 TB per second, and throughput rate of 160 MB per second, the LTO6 has many of the features of the LTO5 and offers support for SAS and FC interfaces. Like the LTO5, it comes in full- and half-height platforms, as single drives, or as part of an automation environment. IT departments can choose the model that makes the most sense for the operating environment, application, capacity, and performance. Oracle's scalable tape automation solutions are designed to accommodate whatever drive and interface is selected. Reference Architecture Components in Tape Environments In environments using tape for backups, the primary logical components of the Oracle Optimized Solution for Backup and Recovery include software that fulfills the following three roles. The software used can be Oracle Secure Backup or other third-party backup software selected by IT staff. Backup software administrative server. Each administrative domain must have exactly one administrative server. The administrative server is configured with complete data for the other hosts in the domain, their roles, and their attached tape devices. This data is maintained in a set of configuration files stored on the server. The administrative server runs the scheduler, which starts and monitors each backup job. The scheduler keeps a backup catalog with metadata for all backup and restore operations performed in the administrative domain. Sun x86 systems or SPARC T-Series servers are recommended for use as administrative servers in this solution because they deliver outstanding performance in the smallest possible configuration while providing considerable room for growth. For optimal performance and availability, the administrative and media server roles should not be mixed within the same server. Backup software media server. A media server is a host with at least one tape drive or library attached to it. The media server transfers data to or from a volume loaded on one of the attached tape devices. Single media servers can be attached to multiple tape libraries, and multiple media servers can share attachments to multiple tape libraries. The media server performs the backup operations under the direction of the administrative server, storing all backup data on its connected storage devices. Sun x86 systems or SPARC T-Series servers are recommended as media servers in this solution because they each offer the most I/O bandwidth for their platform type, a feature that dramatically improves media server performance. Both server families also provide a large amount of capacity for most environments. For optimal performance and availability, the administrative and media server roles should not be mixed within the same server. Backup clients. While this solution focuses on backup and recovery for Oracle Engineered Systems and Oracle Optimized Solutions, it can also be used for network backups of any client platform supported by Oracle Secure Backup. Although this solution includes multiple backup administrative and media servers, a single server can be deployed to support both the administrative and media server workloads if desired. Using two systems provides high availability in a highly scalable solution, but the workloads can be combined on a 14

18 single server. Additional media servers and archive devices can be added as needed to address scalability requirements. Third-Party Software Support Any standard third-party backup and recovery software is supported for the solution, as long as the product supports Oracle Solaris 11 or Linux and Sun ZFS Storage Appliances. Some third-party backup software packages that can be used include the following: NetBackup from Symantec CommVault from Simpana NetWorker from EMC The Oracle components in an Oracle backup solution can be linked through the InfiniBand connections built into Oracle Engineered Systems. Third-party solutions are limited to 1 Gigabit or 10 Gigabit Ethernet connections. Without the benefits of InfiniBand networking, backup performance will slow considerably. In addition, some third-party backup solutions require use of features such as de-duplication or replication. These features also can cause backups to slow dramatically, requiring the addition of disk controllers in an attempt to achieve comparable performance to the Oracle and InfiniBand solution. Oracle Secure Backup Providing scalable, distributed backup and recovery capabilities, Oracle Secure Backup reduces costs and backup complexity through integration with different components of the Oracle stack. This integration provides an Oracle end-to-end backup solution for backing up data from disk to tape and eliminates multivendor integration and support issues. The ready-to-use tape management software in Oracle Secure Backup provides centralized administration, support for heterogeneous networks and NAS devices, and flexible scheduling to simplify and automate protection of the entire Oracle environment, including Oracle databases and file system data. Oracle Secure Backup software provides encrypted tape-based data protection, archival, and recovery management. Running on either Sun x86 systems or SPARC T-Series servers with Oracle tape archive devices, Oracle Secure Backup software scales to support enterprise networks. The networks can consist of a large number of clients, high-performing Network Attached Storage (NAS) devices, or high-performing appliance solutions such as Oracle Exadata, Oracle Database Appliance, and Oracle SuperCluster. Oracle Secure Backup includes the media management layer needed to use tape storage with Oracle RMAN, and the tight integration with Oracle RMAN enables Oracle Database and file system data protection via tape. Oracle Secure Backup uses standard tape formats and supports most popular tape drives and tape libraries across SAN, Ethernet networks, Oracle InfiniBand networks, and SCSI environments. Dynamic tape drive sharing ensures maximum utilization of tape drives. Oracle Secure Backup can be installed on distributed UNIX, Linux, and Windows hosts for backups over the network or to locally attached tape devices. NAS devices, however, cannot be used as Oracle 15

19 Secure Backup clients. To backup and restore a NAS device requires the use of Network Data Management Protocol (NDMP) with Oracle Secure Backup. Sun x86 systems or SPARC T-Series servers are used to create the optimal backup and recovery solution that best fits an organization s data center requirements and standards. Both platforms deliver best-in-class network backup performance and best-in-class reliability. The SPARC T-Series servers from Oracle can deliver up to 20 percent better network performance than the Sun x86 server platforms. However, both types of platforms have similar bandwidth offerings in terms of the number of expansion slots. The recommended components in the solution, including servers, SAN, Ethernet and InfiniBand networks, tape library management, tape encryption management, and tape drives, are configured for redundancy. To facilitate interoperability and deliver more stability, the same platform found in the high-performing appliance solutions, such as Oracle Engineered Systems and Oracle Optimized Solutions, should be deployed wherever possible. Backup Best Practices While in-depth coverage of backup best practices for each solution is out of the scope of this document, some recommended best practices are discussed in the following sections, along with information on where to find thorough best practices for each Oracle solution. Best Practices for Accomplishing Recovery Time and Recovery Point Objectives Oracle offers a complete technology stack comprised of hardware and software. Tightly integrated, the multiple layers of the stack contain tools for data protection and backup and recovery. Important tools can be found at the applications, database, operating system, virtualization, and storage levels, as shown in Figure 12. Following best practices and combining these tools effectively can result in an optimal solution for backing up Oracle systems. In fact, by not relying on any single solution, it is possible to create an ideal data protection environment that reduces reliance on conventional backups. Figure 12. Oracle offers data protection tools at various levels of the integrated stack. 16

20 Among the tools from each layer are: Application: Oracle Secure Backup, third-party backup software Database: Oracle Archive Log Mode, Oracle Data Guard, Oracle Flashback Database, Data Recovery Advisor, Oracle RMAN Operating System: Built-in backup and recovery tools, ZFS snapshots, alternate boot environments (these are Oracle Solaris operating system tools). Storage: Oracle Exadata Storage Expansion Racks, Sun ZFS Storage Appliances, StorageTek Tape storage Using a Combination of Software Tools The tools within the different layers of Oracle's stack can be used to help achieve diverse and precise recovery time and recovery point objectives. By using various tools listed below, it is possible to protect valuable enterprise data and production environments without relying on conventional and lengthier backup and restore operations. Oracle Flashback Database and Oracle Data Guard can help IT staff recover from logical errors and database outages for recovery time objectives within 15 minutes to an hour. Oracle Archive Log Mode enables the database to be rolled back to any point in time within the recovery window to meet specific recovery point objectives. Oracle Enterprise Manager Operations Center enables administrators to deploy Solaris and Linux operating systems for fast operating system recovery. The default file system in Oracle Solaris 11 is ZFS, which supports snapshots, cloning, and replication. This provides the ability at the operating system level to utilize built-in commands for snapshots, cloning, and replication. The Sun ZFS Storage Appliance supports snapshots, cloning, and replication for specific recovery points. Administrators can take a snapshot of an iscsi LUN on the Sun ZFS Storage Appliance to create a backup. Alternatively, deploying a remote Sun ZFS Storage Appliance enables recovery in seconds by switching from the primary to the remote appliance. Best Practices for Backup and Recovery in Virtualized Environments or with Unstructured Data When performing backup and recovery in virtualized environments or with unstructured data, traditional approaches consist of taking disks and applications offline and writing from disk to tape. In the real world, however, organizations require 7x24 availability and cannot afford to take applications offline at all. Oracle Solaris 11 uses Oracle Solaris ZFS (ZFS) as the default file system and ZFS provides features for capturing a system snapshot that can be sent to an external Sun ZFS Storage Appliance or stored on tape. The Oracle Optimized Solution for Backup and Recovery is configured with a Sun ZFS Storage Appliance that can store the virtualized environments and file system snapshots internally. Oracle 17

21 recommends the use of these snapshots to maintain application availability while obtaining a copy of vital data. Best Practices for Backup and Recovery in Virtualized Environments and Unstructured Data with NDMP NDMP is an open standard protocol for network-based backup of network-attached storage (NAS) devices. It facilitates interoperation of backup software and NAS hardware. Since NAS devices are not intended to host applications such as backup software agents and clients, they cannot be set up as backup clients for software such as Oracle Secure Backup. Instead, an NDMP service is created on the NAS device, using NDMP host types to designate the Sun ZFS Storage Appliance as the backup target. These host types require use of specific path names. Please refer to Oracle MOS Note # for examples. Best Practices for Backup and Recovery of Operating Systems Operating system backups are critical and should be made before and after every significant change to the Oracle SuperCluster software. This includes performing backups before and after completing the following procedures: Installation or reconfiguration of significant Oracle or non-oracle software Reconfiguration of significant operating parameters Before and after each quarterly full stack download patch (QFSDP) for Oracle SuperCluster Oracle Solaris 11 The conventional method for backing up an operating system is to install backup and recovery software and perform backups of the operating system to disk or tape, potentially an inefficient approach in 7x24 environments. In Oracle Solaris 10 and Oracle Solaris 11, the ZFS file system enables bare metal recovery of the operating system using ZFS snapshots, the ZFS commands zfs send and zfs receive, and ZFS stream files. A ZFS stream file is an image of an entire ZFS file system placed into a file on an NFS share. The zfs send command creates a stream representation of a ZFS snapshot that is written either to a file or to a different system using standard output. The zfs receive command creates a snapshot whose contents are specified in the stream that is provided on standard input. If a full stream is received, a new file system is created as well. This functionality enables administrators to send and receive ZFS snapshot data and file systems with these commands. Here is a high-level outline of the steps used to perform a backup and bare metal recovery of the operating system. For details and commands, please see the Oracle SuperCluster Administrator's Guide, or MOS Note # To start, create an NFS share on a Sun ZFS Storage Appliance to store the snapshots. Take a ZFS snapshot of the operating system and send it to a stream file using the zfs send command. 18

22 To perform a restore of the snapshot, boot from the Oracle Solaris boot media. Create a new root pool, or rpool (a ZFS storage pool that contains a bootable ZFS root file system), on the replacement disks. Mount the NFS share. To restore, issue a cat command on the stream file, piping it to the newly created pool name on the local disk with the zfs receive command. This procedure restores the zfs send file from the ZFS stream file back to the new ZFS pool. Using zfs send and zfs receive is the preferred method for restoring a bare metal operating system. The previously recommended method using a deployment system takes approximately eight hours contrasted with 45 minutes using the ZFS stream file method. Backing Up Domains The physical resources within Oracle SuperClusters can be virtualized for greater agility and utilization. There are two different methods that can be used for virtualization. The first method creates domains, physically partitioning the hardware to use specified physical resources and physically isolating one domain from another. The second virtualization method creates zones, logically partitioned resources instead of physically partitioned ones. Implementing zones enables the installation of multiple versions of Oracle Solaris 10 and Oracle Solaris 11 on a single system. Various domain configurations and implementations are supported on Oracle SuperClusters. Domain configurations can have different layouts due to the ability to install the domains on different disks. Disk resources can be allocated and shared among domains, while some domains might not own any disks. Disks can be shared from other file systems and appear to users as local hard drives. Single domain systems are typically fairly simple, as are two domain configurations. When backing up these domains, the backup methods are the same as for the operating system. The restoration procedures vary, depending on what devices are installed with the domains, what disks are designated as targets, and where the rpool is created to receive the restore. Domains, or LDOMs, also can be backed up with block-level backups using NDMP and a Sun ZFS Storage Appliance. Note, however, that this method is limited to backing up entire datasets, and does not offer the ability to restore single files. Single file backup and recovery can be accomplished by installing Oracle Secure Backup or third-party backup and recovery software in the zone to backup the zone as a regular backup and recovery client. Backing Up Oracle Solaris Zones Zones typically contain applications or databases with different use cases. There are two different ways to store the zones on the Oracle SuperCluster internal drives, or on iscsi LUNs. Backup and recovery of applications is supported in NFS, and backup and recovery procedures remain the same for snapshots and replication of NFS shares, as well as for tape using NDMP. To backup NFS shares, utilize the Sun ZFS Storage Appliance with NDMP. NDMP, sometimes referred to as dump mode, provides high-performance, block level backups. Note, however, that the ZFS NDMP 19

23 functionality does not permit restoration of individual files, so in order to restore a single file, the entire dataset must be restored. For more information on using NDMP with the Sun ZFS Storage Appliance, refer to the Oracle white paper NDMP Implementation Guide for the Sun ZFS Storage Appliance. iscsi zones can not be backed up at the dump level with the Sun ZFS Storage Appliance they must be backed up in ZFS block mode. This is a high-performance method that works well for restoring the entire zone. The ZFS block mode method does not support restoration of single files or directories the entire dataset must be restored. Single file backup and recovery can be accomplished by installing Oracle Secure Backup or third-party backup and recovery software in the zone to backup the zone as a regular backup and recovery client. For disk-only environments, zones can be backed up through the creation of snapshots, clones, or replication to another Sun ZFS Storage Appliance. Best Practices for Database Backups (Oracle Database 11gR2 or Higher) The following sections outline database backup best practices designed for high availability, optimal performance, and maximum data protection. These best practices should be followed for Oracle Databases running on Oracle SuperCluster, Oracle Exadata, or Oracle Optimized Solutions. Oracle recommends the following schedule for backups of the Oracle Database: Perform weekly Oracle RMAN level 0 (full) backups of the database. Perform daily cumulative Oracle RMAN incremental level 1 backups of the database and use block change tracking. Roll incremental backups into full backup and delay by 24 hours. Perform daily backups of the Oracle RMAN catalogs. Always Use a Fast Recovery Area One vital database backup and recovery best practice is to deploy the database with a Fast Recovery Area (FRA), the location where the database stores all of its information for Oracle Flashback Technologies, snapshots, database backups, rollbacks, and more. The FRA enables the database to protect itself, rolling the database back to a certain point in time and restoring it within a matter of seconds instead of minutes or hours using conventional backup and recovery methods. Consequently, database recovery time is limited by the speed of the device on which the FRA resides, and should be located on the fastest performing storage possible. Oracle Engineered Systems are designed with the FRA built in, since placing the FRA on anything but Oracle Exadata internal storage or an external Oracle Exadata Storage Expansion Rack will lengthen the amount of time it takes for recovery. Additional Database Configuration Best Practices There are other configuration parameters for Oracle Databases that must be set in order to ensure optimal backup performance: 20

24 Set the initialization parameter _file_size_increase_increment= to optimize the space used when incremental (level 1) backups are taken on the FRA. Reset or remove the initialization parameters _backup_ksfq_bufsz and _backup_ksfq_bufcnt on systems running Oracle Database release or later releases. Use a backup Oracle Net service to run against all database servers in the cluster for better performance and high availability. The Oracle RMAN BACKUP command uses the service to automatically spread the backup load evenly among target the instances offering the service. Set DB_RECOVERY_FILE_DEST_SIZE to bound space in the FRA. It is important that the value of this parameter be set to less than the total free space in the disk group, which must take account of at least one disk failure and preferably one Oracle Exadata Storage Cell failure. If multiple databases are sharing the FRA, ensure that the sum of the space allocated to the different databases is less than the free space in the disk group. Consider using Oracle Database Resource Manager to manage system resources, particularly I/O, on the Oracle SuperCluster. Oracle RMAN Configuration Best Practices Parallelize backups across all database nodes, allowing all the disks, network connections, and system CPUs to be leveraged for increased performance. Follow configuration instructions according to whether the backup targets are disks or tapes. Use Oracle RMAN backup scripts to automate weekly and daily backups. Use two to eight Oracle RMAN channels per instance when performing backups to disks. When performing backups to tape, configure one Oracle RMAN channel per tape drive and add tape drives to scale backup rates. Use Oracle RMAN incremental backups and block change tracking. To reduce backup time and resources, perform nightly incremental backups to the FRA and merge them into the image copy backup on regular basis. If recovery is needed, then the copies can be directly used as normal data files and recovered to a consistent point, without the need for a restore operation, thus significantly reducing overall recovery time. Enable Oracle RMAN block change tracking for fast incremental backups. This allows Oracle RMAN to avoid scanning blocks that have not changed when creating incremental backups. The Oracle Exadata Storage Server also offloads block inspection from the database servers. Block change tracking is of greatest benefit for databases where fewer than 20 percent of the blocks are changed daily. For block change rates greater than 20 percent, testing is recommended to ensure that backup times are reduced. Set Oracle RMAN configuration setting FILESPERSET=1 when performing incremental backups to specify the maximum number of files in each backup set. A setting of 1 will allow for a faster single file database restore operation. 21

25 Use an external Oracle RMAN recovery catalog repository as long as it is located on a system that is configured to be highly redundant. Restore and Recovery Best Practices Oracle engineers performing testing in Oracle labs have discovered that optimum restore performance depends on proper management of the backup chunk size. While the chunk size setting is not important during backups, the restore operation must have Oracle RMAN backups broken into the best-performing chunk sizes for individual environments or performance can suffer. Oracle recommends running tests in each individual environment with the actual data to verify that the chunk size to be used for Oracle RMAN restores is going to provide optimal performance. The layout of each individual database determines performance. In addition, to increase recovery rate performance: Create a restore service that runs across all available database nodes. The service is used by the Oracle RMAN restore command. Oracle RMAN automatically balances the restore load among the targeted instances. Best Practices for Performing Local Disk-Based Backups with Oracle RAC Scale backup rates written to local disk in the FRA on an Oracle Database Appliance by using an Oracle RAC configuration. Use multiple instances and start with one Oracle RMAN channel per instance. Continue to add Oracle RMAN channels for performance per instance. Optimal backup rates were observed with all RAC instances and one to four Oracle RMAN channels. Best Practices for Performing Local Disk-Based Backups with Oracle RAC One Node Scale backup rates written to local disk in the FRA on an Oracle Database Appliance by using a single instance and Oracle RAC One Node configuration. Start with one Oracle RMAN channel. Continue to add Oracle RMAN channels to the single database instance to increase performance. Optimal backup rates were observed with two to four Oracle RMAN channels. Database Best Practices for Disk-Based Backups Disk-only backups provide faster recovery times for data and logical corruptions and some tablespace point-in-time recovery (TSPITR) scenarios. Backups to disk also provide the ability to use backups directly with no restore needed by switching to a copy of the database, tablespace, or data file. Best practices for disk-based backup configurations include the following: Scale backup rates for disk: Use all instances and start with two Oracle RMAN channels per instance. Continue to add another two more Oracle RMAN channels for performance. 22

26 Utilize the automatic configuration of the Oracle Exadata Storage Server grid disk group layout during deployment for better performance. This approach assigns the faster (outer) 40 percent of the disk to the DATA area and the slower (inner) 60 percent of the disk to the fast recovery area (RECO) area. An alternative strategy is to purchase additional SATA Oracle Exadata storage specifically for storing the FRA. This allows the application to leverage the full Oracle SuperCluster storage grid, allows the use of lower-cost storage for backups, and provides better failure isolation by using separate backup hardware. To reserve more space and bandwidth for the DATA disk group, Oracle recommends using a tape-based backup solution, or at the very least, a hybrid approach where full database backups are written to tape and incremental disk backups are written to the FRA. Configure a high redundancy DATA disk group to contain the Oracle Cluster Registry (OCR) file, Oracle Cluster voting disk, spfiles, data files, redo log groups, and control files for any disk-based backup solution. Database Best Practices for Tape-Based Backups In addition to the general best practices for Oracle Database backups mentioned above, best practices for tape-based backups include the following: Perform daily backups of the Oracle Secure Backup or third-party backup and recovery software catalogs. Keep the number of archive logs to a minimum to avoid adversely impacting backup rates. Remember that heavy loads on an active database and fully consumed CPUs will affect backup rates. Use Oracle Secure Backup for fast, low-cost tape backups. Oracle Secure Backup provides the fastest database backup to tape due to its tight integration with Oracle RMAN. Using Oracle Secure Backup enables the unused-block optimization capability. However, if the backup is made directly to tape using a third-party media management product, then this does not have any effect because the unused-block optimization is available only with Oracle Secure Backup. Configure the Network Time Protocol (NTP) daemon on Oracle Secure Backup servers. This step ensures that the NTP daemon service on the Oracle Secure Backup Administration and Media Servers are running and configured to use the same time source as the Oracle engineered system. Configure the Preferred Network Interface (PNI) to direct the Oracle Secure Backup traffic over the InfiniBand network interface. This parameter must be set when using a dedicated backup network. Configure one Oracle RMAN channel per tape drive and add tape drives to scale backup rates. Backup performance scales when more tape drives and Oracle RMAN channels are added, assuming there is available throughput on the media server. Configure dedicated Gigabit Ethernet, 10 Gigabit Ethernet, or InfiniBand. Using a dedicated interface for the transport or Sun ZFS Storage Appliance eliminates the impact on the client access network. Use InfiniBand for the best backup rates, especially for larger databases that require fast 23

27 backup rates and low CPU overhead. When not using InfiniBand, use Gigabit Ethernet for smaller databases and 10 Gigabit Ethernet for larger databases. Configure an Oracle RAC service for backup running on all database instances. This step reduces CPU utilization on the database and media server nodes and load balances the backups. Use SQL*Net service load balancing to distribute Oracle RMAN channels evenly among the allocated instances. Tune the network communication when using a third-party media management vendor, and check with the vendor they should also be able to provide configuration best practices. Best Practices for Tape-Based Backups In addition to the database best practices for tape-based backups mentioned above, there are some additional general best practices to consider when performing tape-based backups. Network Configuration Best Practices Care must be taken when configuring the network connections for tape-based backups. The following two sections detail configurations for connecting the media servers to the network. InfiniBand Network to Media Server Configuration Best Practices When connecting media servers to the InfiniBand network for tape-based backups: Directly connect media servers to the InfiniBand fabric by adding an InfiniBand Quad Data Rate (QDR) host channel adapter (QDR HCA) to the media server. Connect the HCA to two different Oracle SuperCluster InfiniBand switches to eliminate the switch as a single point of failure and provide high availability. This configuration ensures transparent failover if connectivity is lost to one of the ports. Configure bonding or IP network multipathing (IPMP) for the InfiniBand interfaces on the media server. Update OpenFabrics Enterprise Distribution on the media server. Configure IP over InfiniBand (IPoIB) connected mode for best performance. Configure MTU Size=65520 on InfiniBand for faster data transmission. Configure the media server to use the InfiniBand network. Gigabit or 10 Gigabit Ethernet Network to Media Server Configuration Best Practices The media servers for tape-based backups can be connected with one or more Ethernet ports that are either Gigabit Ethernet or 10 Gigabit Ethernet. Oracle Solaris enables link aggregation control protocol (LACP) bonding of interfaces for higher network performance. Alternatively, administrators can use Oracle Solaris to configure two interfaces with IP network multipathing (IPMP) for higher availability. LACP bonding requires a 24

28 switch that can support LACP and enables the connection of multiple Ethernet ports to different switches, but having the same Ethernet address. IPMP supports multiple interfaces on the same IP link, connected to a single switch. This is a highly redundant configuration and increases availability, but does not provide high performance. This functionality is only offered by Oracle Solaris. Configure the switch configuration, using Gigabit Ethernet or 10 Gigabit Ethernet on the database server to create either a dual-port Gigabit or 10 Gigabit Ethernet configuration that supports higher backup rates. Configure Gigabit or 10 Gigabit Ethernet on the media server. Configure Persistent Bindings for Tape Devices It is very important that the environment maintains consistent device addresses. In SAN environments running Oracle Linux, persistent bindings must be configured so that the device address does not change. If the device address changes, the media servers cannot access the device until the device configuration within Oracle Secure Backup is updated. Please see MOS Note # for an example of creating persistent bindings for device attachments. Back Up the Oracle Secure Backup Catalog The Oracle Secure Backup catalog maintains backup metadata, scheduling, and configuration details for the backup domain. Just as it is important to protect the Oracle RMAN catalog or control file, the Oracle Secure Backup catalog should be backed up on a regular basis. In Oracle Secure Backup, the catalog backup has been pre-configured: Media family: OSB_Catalog_MF writes all catalog backups to same tape or tapes. Job summary: OSB-CATALOG-SUM sends showing a daily report status of catalog backup to users. Dataset: OSB-CATALOG-DS defines all directories and files to backup for file system backups. Schedule: OSB-CATALOG-SCHED shows the schedule for the catalog backup. The primary catalog backup configuration settings have been defined with only one step remaining which requires user intervention. Edit the OSB-CATALOG-SCHED triggers to specify when to perform the backup. For detailed and up-to-date information, please refer to MOS Note # Best Practices for Oracle Engineered Systems Backups The following sections detail best practices for backing up Oracle Engineered Systems along with pointers for more detailed information. 25

29 SuperCluster Backup Best Practices The recommended backup solution for some Oracle SuperCluster and Oracle Optimized Solutions environments complements the Sun ZFS Storage Appliance Solution for Database Disaster Recovery by adding tape storage and tape storage management. Adding tape storage and tape storage management extends onsite backup storage capacity and enables the creation of off-site backup copies for long-term retention. Oracle recommends making tape backups of primary and remote site data using Oracle RMAN backups and NDMP on a Sun ZFS Storage Appliance through Oracle Secure Backup media servers. For detailed procedures on performing a tape backup of Oracle SuperCluster, refer to the Oracle white paper Protecting SPARC SuperCluster Tape Backup with Symantec NetBackup. Backing Up the Oracle SuperCluster Infrastructure The Oracle SuperCluster infrastructure consists of switch configurations for the InfiniBand fabric. These settings are configured when the system is first deployed and should be backed up to disk or tape in case of human error or a disaster. To backup the switch configuration settings, copy the settings to disk, and then perform an NDMP backup from the internal Sun ZFS Storage Appliance to tape or replicate it to another Sun ZFS Storage Appliance. To create a disk backup of the switch configuration settings, replicate the copy to another Oracle SuperCluster or an external Sun ZFS Storage Appliance. To recover switch configurations, perform a copy back from tape to the internal Sun ZFS Storage Appliance, and from the appliance to the Oracle SuperCluster. For More Information For more information, please see the Oracle white paper Oracle Optimized Solution for Backup and Recovery at or refer to Oracle MOS Note # Oracle Exadata Backup Best Practices The Oracle Maximum Availability Architecture (MAA) group develops Oracle best practices blueprints for Oracle Exadata backup and recovery. The group has published a white paper, Backup and Recovery Performance and Best Practices for Exadata Cell and Oracle Exadata Database Machine, that details best practices for backup and recovery of Oracle Exadata systems. The paper contains performance results for various Oracle Exadata configurations using both disk and tape backups. With detailed steps necessary for configuring the database, Oracle RMAN commands, and network settings for optimal backups, the white paper also includes recommended strategies for restore and recovery operations. In addition, readers can learn more about offloading backups using Oracle Data Guard and gaining additional benefits through the use of Oracle Active Data Guard. Oracle Exadata systems can be backed up to tape or disk. While both media types provide extremely fast backup and restore rates, there are advantages to each. Tape-only solutions isolate faults from the Oracle Exadata Storage Servers and maximize Oracle Exadata capacity and bandwidth. Disk-only 26

30 backups provide faster recovery times for data and logical corruptions and some tablespace point-intime (TSPITR) scenarios. Disk backups also provide the ability to use backups directly with no restore by switching to a copy of the database, tablespace, or data file. Oracle Database Appliance Backup Best Practices As with the other Oracle Engineered Systems, both disk and tape backups provide extremely fast backup and restore rates, but there are advantages to each. Tape-only solutions isolate faults from the Oracle Database Appliance and maximize appliance capacity and bandwidth. Disk-only backups provide faster recovery times for data and logical corruptions and some tablespace point-in-time (TSPITR) scenarios. Backups to disk also provide the ability to use backups directly with no restore by switching to a copy of the database, tablespace, or data file. Configuring the Oracle Database Appliance for Local and External Backups Choosing external versus local backup in the Oracle Database Appliance Manager during deployment will affect the size of the diskgroups +DATA and +RECO, which will determine the FRA size. When configuring the appliance, selecting the external backup type option in the Oracle Database Appliance Manager Configurator utility assigns 80 percent of the disk to the DATA area and 20 percent of the disk to the RECO area. Selecting the local backup type option in the utility assigns 40 percent of the disk to the DATA area and 60 percent of the disk to the RECO area. For More Information For a more detailed discussion of the best practices for backing up Oracle databases running on Oracle Database Appliance, refer to the Best Practices for Oracle Databases section in this paper. The Oracle white paper Backup and Recovery Best Practices for the Oracle Database Appliance contains details on recommended architectures, suggested configuration parameters, best practices for backup and recovery of Oracle Database Appliances, and a detailed section on backing up appliances to networkattached storage (NAS) devices. Also included are performance results for Oracle Database Appliance configurations using local or external disks as backup targets, and detailed steps necessary for configuring the various appliance components in order to obtain optimal backup results. For more information, please see the white paper, or refer to MOS Note # For detailed information on tape backups, refer to the Oracle white paper Protecting Oracle Database Appliance Tape Backup with Oracle Secure Backup. The paper can be found at this URL: Best Practices for the Oracle Sun ZFS Backup Appliance Oracle recommends certain procedures for optimal configuration and performance of an Oracle Sun ZFS Backup Appliance with Oracle SuperCluster or Oracle Exadata. Some of these procedures are outlined here. 27

31 Network Configuration Best Practices Network configurations for the Sun ZFS Storage Appliance vary according to the environment where the appliance is deployed. If the Sun ZFS Storage Appliance is to be used to backup one or two Oracle engineered systems and it is located within 100 meters of the Oracle engineered systems, then InfiniBand networking can be used to connect all the systems. With this recommended configuration, each Oracle engineered system resides on its own InfiniBand fabric and different InfiniBand subnet. When used with an Oracle SuperCluster, the Sun ZFS Backup Appliance can be connected directly to the Oracle SuperCluster InfiniBand leaf switches if the Sun ZFS Backup Appliance is the only other appliance or device (other than Oracle Exadata Storage Expansion Rack) that will be connected to the infrastructure. The Sun ZFS Backup Appliance can be connected to external InfiniBand leaf switches if more appliances or devices will be connected. If the Sun ZFS Storage Appliance is to be used to backup more than two Oracle engineered systems, or the Sun ZFS Storage Appliance cannot be located within 100 meters of the Oracle engineered systems, then 10 Gigabit Ethernet should be used to connect the Oracle engineered systems to the Sun ZFS Storage Appliance. Network Configuration for InfiniBand Connectivity To provide the highest availability and performance, Oracle recommends that each Sun ZFS Storage Appliance controller be configured with two Dual Port QDR InfiniBand HCA cards, providing a total of four ports per Sun ZFS Storage Appliance controller. The controllers should also be connected to a pair of redundant 10 Gigabit Ethernet switches to provide continuous connectivity in the event of an outage. This configuration requires the use of an Active/Standby IPMP group. For configuration details, see the Oracle white paper Configuring a Sun ZFS Backup Appliance with Oracle SPARC SuperCluster. Network Configuration for 10 Gigabit Ethernet Connectivity To provide the highest availability and performance while not using the InfiniBand network, Oracle recommends that each Sun ZFS Storage Appliance controller be configured with two Dual Port 10 Gb Optical Ethernet cards, providing a total of four ports per Sun ZFS Storage Appliance controller. This configuration requires the use of an Active/Standby IPMP group. Additionally, for performance reasons, the 10 Gigabit Ethernet network should be configured with a large MTU size also known as jumbo frames. For configuration details, see the Oracle white paper Configuring a Sun ZFS Backup Appliance with Oracle SPARC SuperCluster. Configuring Network Cluster Resources Once the network resources have been configured for either 10 Gigabit Ethernet or InfiniBand, the interfaces need to be placed under control of the Sun ZFS Storage Appliance cluster. Clustering the network resources provides continuous access to the data on the Sun ZFS Storage Appliance in the event of a controller head failure in the appliance. 28

32 Configuring Networking for the Sun ZFS Backup Appliance The InfiniBand ports on the Sun ZFS Backup Appliance must be configured for IP multipathing. Four IP addresses will be required for each Sun ZFS Backup Appliance head for a total of eight addresses since the interfaces will be running in an active-active configuration. For More Information For more detail, and a list of principles to use for database restore operations, see the Oracle white paper Backup and Recovery Performance and Best Practices Using the Sun ZFS Storage Appliance with the Oracle Exadata Database Machine. Please refer to MOS Note # to get details on the recommended software stack for the Sun ZFS Storage Appliance when used with Oracle Exadata. Refer to the Oracle white paper Configuring a Sun ZFS Backup Appliance with Oracle SPARC SuperCluster for more details on how to configure an Oracle Sun ZFS Backup Appliance for use with a Oracle SuperCluster to provide maximum throughput and availability. Storage Configuration Best Practices Configuring the Sun ZFS Backup Appliance storage pools assigns physical disk drive resources to logical storage pools for backup data storage. When backing up and recovering the Oracle Database to the Sun ZFS Storage Appliance, Oracle recommends that the Sun ZFS Storage Appliance disk resources be configured in two equally sized pools. Configure the two pools by assigning half of the physical drives in each drive tray to each storage pool to maximize system throughput. Once the pools have been created, they should be placed under the control of the Sun ZFS Storage Appliance cluster to protect against the failure of a network component supporting the appliance or a component within the appliance controller itself. Doing so ensures that an Oracle RMAN database backup or restore operation can continue to run. For more details, see the Oracle white paper Backup and Recovery Performance and Best Practices Using the Sun ZFS Storage Appliance with the Oracle Exadata Database Machine or the Oracle white paper Oracle Optimized Solution for Oracle SuperCluster Backup and Recovery. Configuring the Sun ZFS Backup Appliance Oracle RMAN Projects and Shares Share configuration is the process of setting up and running NFS mount points for client access. A project is an entity that provides a higher-level management interface point for a collection of shares. Two projects should be created for the Oracle Database 11g R2 Oracle SuperCluster configuration one project per pool. To optimize share management, update the default mount point for shares contained in the project to reference the database name. To optimize a system for performance, create four shares for each project in each pool, for a total of eight shares (four on each head). Oracle provides an Oracle Exadata utility downloadable from Oracle Technology Network (OTN) at the Sun NAS Storage Downloads page that automates the process of creating Sun ZFS Storage Appliance Projects and Shares and creates the corresponding entries on the database nodes in the cluster. The utility also creates Oracle RMAN command files that can be used for backing up the database. Running the utility ensures that all of the best practices of using the Sun ZFS Storage Appliance as an Oracle Exadata backup destination are followed. 29

33 Create two projects, one per pool. Configure the projects to use a write bias of throughput. Configure the projects NFS Exceptions to allow access to the shares by the Oracle engineered system. Create at least eight shares per project for optimal backup and restore rates. Database Backup and Restore Key Best Practices with a Sun ZFS Storage Appliance and Oracle Engineered Systems The following key practices should be implemented in order to obtain maximum availability and performance from the Sun ZFS Storage Appliance. Note, however, that for any backup operation or changes to backup practices, IT staff should perform testing to ensure that the backups complete within the available backup window and that the database restore and recovery operations can meet recovery time objective requirements. Configure Network Connectivity Enable Direct NFS. The Oracle Database running on an Oracle engineered system is backed up to a Sun ZFS Storage Appliance using the Oracle Database version 11g Release 2 Direct NFS option. Note, however, that by default, the Direct NFS option is not enabled, and enabling Direct NFS requires a database shutdown in order to relink the Oracle kernel. By enabling Direct NFS, the operating system kernel NFS processes are bypassed and the Direct NFS utility opens up 1 MB network buffers solely for use by Oracle RMAN, bypassing the system configured TCP network buffers, and speeding up the database backup and restore. For details, see the Oracle white paper Backup and Recovery Performance and Best Practices Using the Sun ZFS Storage Appliance with the Oracle Exadata Database Machine or the Oracle white paper Oracle Optimized Solution for Oracle SuperCluster Backup and Recovery. Configure oranfstab. It is not essential to create an oranfstab file for the Oracle Database to utilize Direct NFS. However, when using the following configurations, an oranfstab file must be created to fully utilize the network resources between the database nodes in Oracle Exadata or Oracle SuperCluster and Sun ZFS Storage Appliance. An Oracle engineered system is to be connected to the Sun ZFS Storage Appliance using InfiniBand OR An Oracle engineered system is running Oracle Solaris OR An Active/Active IPMP group is created on the Sun ZFS Storage Appliance Refer to MOS Note # , RMAN Backup From SPARC SuperCluster to Sun ZFS Backup Appliance, for sample /etc/oranfstab files. 30

34 Follow Recommended Best Practices for Database Backup and Restore Oracle recommends using a combination of Level 0 and Level 1 backup sets when backing up the Oracle Database to the Sun ZFS Storage Appliance. Frequency of backups is dictated by recovery time and recovery point objective requirements. Note that when using the Sun ZFS Storage Appliance, Oracle does not recommend an incrementally updated backup strategy using a combination of data file copies and incremental backup sets that are subsequently merged into the data file copies. When an incrementally updated backup strategy is selected, the backup solution becomes I/O bound during the merge process, and alleviating this bottleneck requires a significant number of available disk spindles in order to achieve the required IOPS rate. For most IT departments, the following recommendations should be sufficient. The Oracle Database FRA should remain on the Oracle Exadata Storage Servers that are part of the Oracle Exadata or Oracle SuperCluster and should not be put on the Sun ZFS Storage Appliance. The weekly full level 0 backup should be written to the Sun ZFS Storage Appliance. A daily incremental level 1 backup should be written to the Sun ZFS Storage Appliance. A maximum of 16 Oracle RMAN channels should be allocated and configured for both weekly Level 0 and daily Level 1 backups to write to one of the 16 Sun ZFS Storage Appliance shares created using the Oracle Exadata Backup Configuration Utility mentioned earlier (or created manually). Two separate backup jobs should be submitted if Oracle RMAN compression is used to preserve space on the Sun ZFS Storage Appliance but the database consists of a mixture of uncompressed data and compressed data or data that cannot be compressed further. One of the backup jobs will create an uncompressed backup, and the second a compressed backup. Otherwise, compressing data that does not typically yield good compression lengthens the backup window significantly, consumes unnecessary CPU resources, and results in very little space savings. Oracle RMAN offers an option to amortize the backup over a given time period and minimize the performance impact to critical applications. For example, if there is an eight-hour backup window available for the weekly level 0 backup of a 20 TB database, but critical database functions are still running during that backup window, the Oracle RMAN duration minimize load option can spread the work over the entire backup window. The use of Oracle RMAN options such as sectionsize is recommended. These options are included in the backup scripts generated by the Oracle Exadata Backup Configuration Utility. The sectionsize parameter breaks up BIGFILE tablespaces into more manageable amounts. Oracle recommends that the largest tablespaces be broken up into equal size sections for each Oracle RMAN channel. For instance, if there are two tablespaces of 10 TB and 14 TB, and there are 16 Oracle RMAN channels, then the sectionsize parameter should be set to approximately 320 GB. The Oracle Exadata Backup Configuration Utility does not enable Oracle RMAN or Sun ZFS Storage Appliance compression, as this is something that is dependent on the data to be backed up and the database licensing options purchased. Enabling Oracle RMAN compression increases CPU utilization. 31

35 Tune the Oracle Database Instance for Oracle RMAN Optimizing high-bandwidth backup and restore operations using Oracle RMAN and the Sun ZFS Storage Appliance requires adjusting the instance parameters that control I/O buffering. For information about how to tune these parameters, see Article ID : RMAN Performance Tuning Using Buffer Memory Parameters at Configure Oracle RMAN Performance benefits can be realized by configuring 16 or 32 Oracle RMAN channels to be evenly distributed over the Oracle Database instances and nodes in the Oracle RAC cluster. The channels also should be evenly distributed over the shares exported from the Sun ZFS Backup Appliance. Best Practices Using an External Sun ZFS Storage Appliance or Oracle Exadata Storage Expansion Rack Traditional backups are performed with a base-level full backup and then backups at subsequent intervals to capture only the files that have changed. This can be an inefficient method since restore operations must start with the full backup and apply each incremental in the proper order. Also, any and all incremental backups must be applied or the restore will not be complete. When using external Sun ZFS Storage Appliances or Oracle Exadata Storage Expansion Racks for backups, it is possible to follow a particular strategy as shown in Figure 13 below. Backup operations should start with a full Level 0 backup on the first day. On the second day, IT staff can perform an incremental (Level 1) backup to record just the blocks that have changed from the previous day, or they have the option to apply the Level 1 incremental to the full Level 0 backup as shown in Figure 14 (using Day 3 as an example). The incremental applied backup merges any changes since the full backup and creates a single restore point. This is a much more efficient method than the traditional combination of full and incremental backups typically used to obtain an up-to-date restore. Figure 13. Oracle suggested backup strategy for databases. 32

36 Use ZFS Snapshots The above procedure can be taken a step further by utilizing the snapshot capabilities of the Sun ZFS Backup Appliance as illustrated in Figure 14. It starts again with a full Level 0 backup on Day 1 and an incremental backup on Day 2. On Day 3, the incremental backup is applied to the Level 0 backup from the first day, and a snapshot is taken of the applied incremental, resulting in an image that looks like a version of a full backup. This consumes a lot less space because it is only recording changes. Figure 14. Oracle suggested backup strategy with Sun ZFS Backup Appliance. Using Non-Exadata Storage for Oracle SuperCluster and Oracle Exadata Database Machine Backups While not recommended, non-exadata storage solutions might be necessary for users in some cases. Differences in network connectivity, storage configuration, and protocols used by non-exadata storage will affect backup performance and contribute complexity. The MAA white paper contains suggestions on how to avoid the pitfalls of non-exadata storage in these environments. For more information and a listing of best practices, please refer to this URL: For configuration details and up-to-date changes, please see MOS Note # For more information on Oracle's Maximum Availability Architectures, please see the following resource: 33