WHITE PAPER. Oracle RMAN Design Best Practices with Data Domain. Storage. Deduplication

Transcription

1 WHITE PAPER Deduplication Storage Oracle RMAN Design Best Practices with Data Domain w w w. d a t a d o m a i n. c o m

2 DATA DOMAIN I Contents Contents Integration...9 Introduction Basic Concepts Additional concepts....3 Executive Summary....4 Oracle Background...4 Terminology...5 Types of backups RMAN definitions ASM...8 Planning...9 Important RMAN options Oracle Recommendations Purpose...12 Questions and Answers...12 Summary Appendix A Troubleshooting Using Oracle Enterprise Manager From the command line Data Domain Product Background Models Software Releases....8 Deployment options...8 How it works Advantages of Data Domain in an Oracle environment Acknowledgements Much of the information contained herein is taken from copyrighted Oracle documentation that is publicly available. All rights by Oracle to any of this documentation is, of course, retained and acknowledged. 2

3 Introduction This technical note gives information on the configuration, setup and tuning of Oracle RMAN 9.x/10.x with Data Domain deduplication storage systems. It is expected that the reader of this paper has a passing knowledge of both Oracle management practices as well as familiarity with backup and recovery techniques for Oracle. A working knowledge of UNIX/Linux and/or Microsoft Windows and the networking components of each is also required. Finally a basic understanding of the setup and management of a Data Domain System is also required. This perhaps seems like a great deal of background required but will actually serve more as a shortcut to understand some of the shorthand used in this paper. Some theory and background will be presented, but generally only to further explain the recommendations provided. Oracle s Recovery Manager (RMAN) is a built-in tool that allows the Database Administrator (DBA) to easily backup and recover the data in an Oracle database. RMAN handles the coordination required to ensure that transaction integrity is preserved and sufficient information is maintained to recover the database to any appropriate point. RMAN can create backupsets that comprise as much or as little recovery information as the DBA requires but usually includes information from the database datafiles, controlfiles, and redo logfiles. By default RMAN is set up to backup database files to a tape, a disk, a Windows UNC path or UNIX NFS mount path. There are tuning parameters that can be applied for optimum performance and compression. Example of Traditional Backup Environment Oracle has now released the next generation database, 11g. As acceptance of this new technology grows, this document will be updated and revised with additional information as necessary. For now, it is recommended that 11g users follow the advice for 10g. Clients Oracle Server Primary Storage WAN Basic Concepts A Data Domain System is an appliance which is typically used as a target for backup or archived data. The characteristics that make it good for this include: Support for any conventional nearline application through generalized support for Network Attached Storage (NAS) interfaces over Ethernet, a Virtual Tape Library (VTL) interface option over Fibre Channel, and product-specific interfaces such as NetBackup OpenStorage; High-speed, inline deduplication using small, variable-sized sequences to identify redundancies; Integrated data protection technologies such as RAID-6, postbackup data verification, and periodic validation checks of existing data sets; Deduplicated replication to automate disaster recovery (DR). While a Data Domain appliance offers NAS interfaces, its use as a general purpose file system is generally not recommended because it has not been tuned for small, random-access I/O requests typical of database transactions; it is better tuned for applications that read or write a whole file at a time. On a Windows network, the Data Domain system presents as a network file server accessible via Microsoft s Common Internet Filesystem (CIFS) protocol. To a UNIX or Linux network, the Data Domain system presents as a network file server accessible via NFS. A single Data Domain system can present both interfaces simultaneously. Figure 1 - Example of Traditional Backup Environment Additional concepts Backup/ Media Server Staging Disk Tape Library Offsite Storage Other useful concepts to be aware of prior to getting too far into this paper deal with the details of accessing a Microsoft Windows shared drive, either by mapping a drive a referencing a Universal Naming Convention (UNC) name and setting up and accessing NFS mounts on UNIX/Linux boxes. In either case, sufficient information should be provided during setup to ensure that RMAN has access to the necessary path when it runs. Frequently, user credentials are forgotten during this step so ensure they are correct prior to starting RMAN. Much the same applies to NFS for a UNIX/Linux environment. Appropriate edits to the /etc/fstab or /etc/vfstab files will be required Finally, credential checking also occurs on the Data Domain system. Ensure that these are setup appropriately. 3

4 Executive Summary To save reading for technicians already well briefed on both Oracle and Data Domain, here is a summary of the suggested best practices. Explanations and reasoning for these are included further on. Parameter or option Type of backup FILESPERSET 1 MAXOPENFILES 1 Parallelism or number of Channels CONTROFILE AUTOBACKUP Encryption Compression ( AS COMPRESSED clause) NFS MOUNT OPTIONS Sun Solaris: Linux and other UNIX: Access to mount point: Replication MISCELLANEOUS OPTIONS Use of multiple Data Domain systems to backup very large databases Multiplex archivelogs to the Data Domain system Full Setting As many as the system can bear. Usually ranges between 3 and 5. On No No Setting rw,noac,hard,rsize=32768,wsize=32768, llock rw,noac,hard,rsize=32768,wsize=32768, nolock Restrict access to a mount point to only the UNIX machines actually running RMAN and/ or any standby, recovery systems. Yes, use Data Domain system to replicate backupsets to remote DR site Setting Yes, but be sure to carefully configure channels to ensure the same datafiles are always backed up to the same Data Domain system. Multiple copies of archivelogs provide added protection. Specifying the Data Domain system as an optional destination increases protection and gets the logs to the system even sooner than waiting until the RMAN backup runs. Figure 2 - Summary of Best Practice Settings Clients Oracle Server RMAN Environment with Data Domain Primary Storage Onsite Retention Storage Archive to Tape WAN Figure 3 - RMAN Environment with Data Domain Offsite Disaster Recovery Storage Backup Oracle Server Oracle Background Oracle environments range from the small, experimental installations to the large production environments that run multi-billion dollar corporations. Because of this large difference in scale, a single recommendation is seldom applicable across all environments. However, some basic best practices for maintaining a reliable Oracle database exist 1. Run in archivelog mode In this mode, Oracle does not delete the redo logfiles that track changes to the database. Instead it archives these files to secondary (and sometimes tertiary) locations when Oracle performs its periodic logswitch. Without this history of archived logfiles, recovery of an Oracle database to a point prior to an error is significantly constrained. Keep the production Oracle data on extremely reliable storage Enough has been written about various means for storing Oracle data on SAN or NAS storage that little more need be said here. Spread the Oracle data across many disk devices Similar to the previous item, it is important to follow Oracle s and/ or the storage vendor s documentation for optimal storage of Oracle database files. Perform full Oracle backups on a regular basis. Having multiple recovery options is important to the Oracle DBA as the size and importance of an Oracle database grows. Starting in Oracle 10g, Oracle introduced the Flashback Recovery Area (FRA). This is a special storage area set aside to hold previous copies of the Oracle data to ease and simplify some recovery operations. For the purpose of this paper, we will focus more on the use of RMAN to create backup information outside the FRA that can be used for recovery.it is important that all DBA s develop a practice for doing regular backups, the base of which is a full backup. Periodic practice recoveries are also recommended to ensure all the pieces, processes, and skills are ready and available for the inevitable day when a critical recovery is needed. We will get into the various types and levels of Oracle backups later in this paper. Keep archivelogs and/or incremental backups in between each full backup Keeping a history of archivelogs is critical to maximizing recovery opportunities. A backupset consisting of a full database backup along with all the subsequent archivelogs allows that DBA to recover the database to any point desired. Incremental backups also provide in-between recovery opportunities. However, many DBA s eschew the use of incrementals in favor of nightly fulls if possible since the recovery involving an incremental is a bit more complicated and time consuming. One benefit of storing to deduplicated storage is that when doing nightly fulls, the new storage consumed is typically less than an incremental would require. 1 Many useful documents exist on the Oracle website, too many to be cited here. However, one especially useful document is the RMAN Best Practices which is included verbatim later in this document. 4

5 Keep one or more copies of the data on disk in separate locations Unfortunately we live in an uncertain world, and keeping only a single set of backup files runs the risk that something could happen to make it unavailable or unusable. Therefore it is highly recommended that the backup practice arranges to keep multiple copies of the backup files in various locations. Use RMAN to help organize the various backups and simplify possible recovery To simplify all these different operations, copies, and locations of files, the use of RMAN is highly recommended. It assists the DBA in keeping track of the state of all these various backup images as well as their locations. It also assists in making the decision that particular backup information has become obsolete and can be safely retired. Finally, it makes recovery for the DBA much simpler since RMAN keeps track of all the files required for certain recovery operations the DBA might require. Backupset A backupset is a set of one-or-more files that is written by Oracle as the output of a backup operation. The DBA can characterize a backupset in a number of different ways through different naming facilities as well as a separate tagging facility. What is important to remember is that the files in a backupset can hold interleaved data from a number of different Oracle data files unless an image backup is being performed. In this latter case, RMAN backs up each Oracle datafile to a separate file within the backupset. Terminology Types of backups RMAN supports several different kinds of backups. A full backup, as its name implies, is a backup of the entire database. A partial backup can be of a single or multiple portions of the database but not necessarily everything. An incremental backup uses a facility within Oracle to track changes to the database that occur between backups to backup only those specific pieces of the database that have been altered. The final type of backup is an image backup which makes precise copies of the Oracle database files but the result looks just like a copy of the files when viewed from the operating system. Figure 4 - Example: RMAN list backup command RMAN Parameters Several options exist with RMAN that every DBA should be aware of. These effect how RMAN behaves when performing its backup duties. Some of the most important are outlined here because they affect how RMAN will interact with a Data Domain deduplication storage system. For more complete description of these options, refer to Oracle s documentation. In each of these backups RMAN processes the data to be backed up and consolidates it into one or more files called a backupset. We will spend some time talking about backupsets in just a little while. RMAN definitions Oracle RMAN has a set of definitions and parameters that are unique to it and must be understood by a DBA (and a reader of this paper) to get very far with RMAN. 5

6 Some RMAN parameters are global and persistent, others apply to only specific tasks. A listing of the global parameters can be generated by the show all RMAN command: RMAN> show all; RMAN configuration parameters are: CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default CONFIGURE BACKUP OPTIMIZATION OFF; # default CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO %F ; # default CONFIGURE DEVICE TYPE DISK PARALLELISM 3 BACKUP TYPE TO BACKUPSET; CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT /dd580a/ oracle/%u ; CONFIGURE CHANNEL 1 DEVICE TYPE DISK MAXOPENFILES 1 FORMAT /dd580a/backup/oracle/backup_%u ; CONFIGURE CHANNEL 2 DEVICE TYPE DISK MAXOPENFILES 1 FORMAT /dd580a/backup/oracle/backup_%u ; CONFIGURE MAXSETSIZE TO UNLIMITED; CONFIGURE ENCRYPTION FOR DATABASE OFF; # default CONFIGURE ENCRYPTION ALGORITHM AES128 ; # default CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default CONFIGURE SNAPSHOT CONTROLFILE NAME TO /home/oracle/ oracle/product/10.2.0/db_1/dbs/snapcf_orcl.f ; # default Figure 5 - Example: RMAN show all command Maxopenfiles This parameter controls the number of Oracle datafiles that RMAN can have open for reading at any one time. MAXOPENFILES = integer Controls the maximum number of input files that a BACKUP command can have open at any given time (the default is 8). Use this parameter to prevent Too many open files error messages when backing up a large number of files into a single backup set. An important effect of this parameter is on the amount of system memory required by RMAN while a backup is occurring. To minimize the number of I/O buffers that RMAN allocates it is important to keep MAXOPENFILES to the smallest value possible while still allowing good performance 2. FILESPERSET This option, a parameter on the begin backup RMAN command, tells RMAN how many Oracle data files it can combine into a single backup file. In other words, it controls the multiplexing of Oracle data into the backupsets. FILESPERSET [=] integer When used with commands that create backupsets, specifies the maximum number of files to include in each backupset created. By default, RMAN divides files among backupsets in order to make optimal use of channel resources. The number of files to be backed up is divided by the number of channels. If the result is less than 64, then it is the number of files placed in each backupset. Otherwise, 64 files will be placed in each backupset. Maxsetsize This parameter defines the largest size a single file within the backupset can reach. MAXSETSIZE [ CLEAR TO [sizespec UNLIMITED ] Specifies the maximum size of each backup set created on a channel. By default MAXSETSIZE is set to UNLIMITED, meaning that it is unconstrained. Compression RMAN provides two types of compression. First, and mostly invisible to the DBA is unused-block elimination which was introduced with Oracle 10g. If RMAN encounters a data block in a file being backed up that is not in use, it will not send it to the backup stream at all unless a certain set of conditions exists (see Oracle s RMAN documentation for details). The second type of compression tells RMAN to apply a compression algorithm (BZIP2) to the data being written to the backupset. This results in less disk space being used at the cost of significantly greater CPU consumption during the backup operation. To quote from Oracle s own documentation: Note: If you are backing up to tape and your tape device performs its own compression, you should not use both RMAN backupset compression and the media manager vendor s compression. In most instances you will get better results using the media manager s compression. See the discussion of tuning RMAN s tape backup performance in Oracle Database Backup and Recovery Advanced User s Guide for details. We will discuss settings for compression later in this paper. Encryption RMAN provides for backupsets to be encrypted to increase security. Any backupset can be encrypted except image backupsets. Automatic encryption requires the configuration and setup of key management functions such as provided by the Oracle Encryption Wallet. Channels and parallelism RMAN backs up Oracle datafiles using a series of separate processes (i.e., separate running programs and/or threads) which all run in parallel. Typically, while one process is waiting on I/O to complete for a given set of reads/writes, another process can be performing similar tasks against a separate set of files. By keeping these channels running simultaneously but against data stored on different storage 2 See Oracle MetaLink Note: RMAN: I/O Slaves and Memory Usage 6

7 devices, the overall RMAN backup can complete in less time than if the entire process ran sequentially against a single file at a time. This example explicitly parallelizes a backup operation by specifying to Oracle Enterprise Manager that multiple parallel processes should be used: Figure 6 - OEM Backup Settings By specifying to OEM that Parallelism should be 4 and (on a different screen) that FILESPERSET should be set to 1, the resulting script that OEM will run looks like: Figure 7 - OEM example script 7

8 The DBA can either define a global setting for parallelism and allow RMAN to balance the I/O load as best it can, or the DBA can manually define these channels to get even better performance than Oracle is able to obtain on its own. Channel Allocation described will apply equally well to all backup scenarios and recommendations in this paper hold for all presently available (and currently planned) versions of Data Domain Operating System (DD OS). Deployment options A given Data Domain system is usually deployed in one of two modes; either as a Network Attached Storage (NAS) device or as a VTL. We earlier stated we were not going to cover the VTL deployment. This is because RMAN does not have the capability to talk directly to a VTL; it relies on an intermediate backup application to provide that function. Server Session Server Session channel ch1 channel ch1 Recovery Manager Therefore this paper focuses on the use of a Data Domain deduplication storage system with RMAN when RMAN is using the system directly as either an NFS device or a CIFS device. Both of these options involve attaching the Data Domain system to an Ethernet, giving it an appropriate IP address on the network, and configuring the rest of the network information as required to deploy a typical file server for the given NFS/CIFS environment. It is presumed the reader is already familiar with the procedures for accomplishing this. Figure 8 - Channel Allocation Overall effects Some of these parameters have significant effects on how well Oracle RMAN behaves when backing up to a Data Domain system. This includes the use of the Data Domain system as a direct target for RMAN (either using an NFS mount or a CIFS share) or if the files created are subsequently copied to the Data Domain system using a backup application or a operating system utilities (e.g., cp or dd for UNIX/Linux or copy for Windows). These effects, and combinations thereof, will be discussed in a later section. ASM Oracle s Advanced Storage Manager (ASM) was introduced in Oracle 9i and further enhanced and extended in 10g. For the purposes of this paper, ASM actually is involved very little except in related areas of managing I/O to the Oracle datafiles as efficiently as possible. However, since ASM is a database unto itself, it is important for a DBA to keep in mind the need to maintain effective backup and protection of the information required by ASM to provide its service to the rest of the Oracle environment. Data Domain Product Background Models Data Domain systems are available in a variety of system configurations. Fortunately, these configurations generally all run the same software (with the restriction that newer systems sometimes have to run newer software) so different Data Domain systems models have little or no effect on the operational characteristics within an RMAN environment. Software Releases Much the same can be said of the software releases for the line of Data Domain systems. This software has been designed to be agnostic to the data being stored on it; all data presented is effectively deduplicated, compressed, and protected. The characteristics How it works The Data Domain system runs a proprietary file system that accepts backup data from its network interfaces and writes this data to disk. In the process, it examines the incoming datastreams and eliminates the data segments that have been seen before and writes small segment indicators in their place. It also applies an LZ-type (or optionally GZ) compression algorithm to the data just before writing it to disk. These characteristics, and several others, make the Data Domain system an ideal target for Oracle RMAN. For further understanding on these mechanisms, please refer to the SISL white paper on the Data Domain Web site at Advantages of Data Domain in an Oracle environment By eliminating redundant data segments, the Data Domain system allows many more backups to be stored and managed than would normally be possible for a traditional storage server. While completely new data has to be written to disk whenever discovered, the variable-length segment deduplication capability of the Data Domain system makes finding identical segments extremely efficient. For example, a full backup of an Oracle database will be stored on the Data Domain system in approximately 40% of the space it originally occupied. Why? First, even in a full database backup, some data segments are repeated and eliminated. Not many, but let s say 20%. The data is then compressed using an LZ-compression algorithm to further compress the data which usually gets about 2X compression (or 50% space reduction). So these two numbers generally yield approximately 60% reduction. The next time a full backup is performed, a much higher percentage of the data has already been seen; often well over 90% depending on the database. So even though the DBA requested a full database backup, the Data Domain system only has to write a small fraction of the whole database to disk this time. This effect accumulates 8

9 over time so that the compression ratio continues to improve until a plateau is reached where the new data being written is nearly the same as the amount of old data being retired. A better example is presented later. The integration of the Data Domain system into an Oracle/RMAN environment is quite seamless since the system presents as either a NFS or CIFS shared storage server. Oracle/RMAN already supports and documents this type of installation for effective RMAN storage. The ability of the Data Domain system to store several weeks or months of full Oracle backups provides the DBA a backup and recovery environment with great flexibility and protection while occupying a minimal amount of space and using a minimal amount of resources. For all critical Oracle environments, it has become a best practice to replicate the production Oracle data to a secondary recovery location. The DBA has many options to choose from including tools built-in to Oracle itself, solutions offered by enterprise storage providers, as well as third party solutions. The Data Domain system also offers replication facilities which are extremely easy to deploy. The primary advantage of Data Domain system replication is the fact that the data being moved is all de-duplicated and compressed prior to moving. Also, as the RMAN backup process proceeds, each file in the backupset is rapidly replicated to the remote site as soon as it is closed allowing the overall time-to-dr to be minimized since much of the replication takes place even while the RMAN backup process is still active. Allowing the Data Domain system to provide this service reduces CPU and network requirements that would be required if either Oracle itself or the enterprise storage were to perform similar service. And, of course, the replicated image occupies the same minimal footprint as the primary backup image, keeping overall costs to a minimum. Integration Planning Planning to use a Data Domain system as a target for an RMAN backup is relatively straightforward since the system just appears as disk storage. However, consideration for space required, network throughput, replication effects, recovery operations, etc. must be taken into account to ensure that the entire Oracle + Oracle/ RMAN + operating system + network + Data Domain system fulfills the customer s requirements for backup windows, recovery time objectives, desired retention, and support for disaster recovery. Capacity How much space is required? This is obviously an important question to answer when some of the databases being backed up are growing to multiple terabytes in size. A detailed analysis of this question can be developed by a knowledgeable Data Domain System Engineer who can apply extensive profiles and models developed through thousands of real-world deployments. But a simple approach might go something like this. Database size 500GB Combined size of all Oracle datafiles and logfiles Rate of change 5% per day Involves 100% change of logfiles and 10% change to database files 3. Daily backup size 500GB + 25GB Datafiles + archivelogs Retention period 3 weeks Figure 9 - Sizing considerations With 5% change rate each day, multiplied by 2 to account for the change being in both the datafiles and the archive logfiles (perhaps a bit more in UNDO depending on configuration), the amount of new data presented is about 350GB/week. Assume the combination of de-duplication and local (LZ) compression provides 4x (or about 12.5GB of actual new data needing to be stored), then this example with full daily backups would look like: Day of week Cumulative Amount backed up Cumulative storage consumed Overall compression Sunday 500GB 125GB 4X Monday Tuesday Sunday + 550GB = 1050GB Monday + 550GB = 1600GB 137.5GB 7.5X 150GB 10.5X Wednesday 2150GB 162.5GB 13.2X Thursday 2700GB 175GB 15.4X Friday 3250GB 187.5GB 17.3X Saturday 3800GB 200GB 19X Sunday # GB 212.5GB 20.4X Sunday # GB 300GB 27.3X Sunday # 4 12,050GB 387.5GB 31.1X Figure 10 - Cumulative storage example In this example, the customer now has three full weeks of backups kept on disk with each image being a fully recoverable image. Over 12TB of data has been sent to the Data Domain system but due to the effects of deduplication and compression, the actual space consumed is held to less than 400GB. This example also assumes the database in question does not grow in size which is unlikely, but a reasonable rate of growth would only change the numbers by a small percentage.the space used will start to level out as the retention period for backups is reached and RMAN starts letting them expire. This point is determined by the DBA with various retention settings within RMAN. 3 Of course, Oracle housekeeping will create a great deal of change to the logfiles without necessarily making any changes to the database, so these numbers are unlikely to be terribly precise. 9

10 The space used will start to level out as the retention period for backups is reached and RMAN starts letting them expire. This point is determined by the DBA with various retention settings within RMAN. Of course, the best and most accurate way to make a determination on the required storage and the expected compression effects would be to do a pilot using real customer data. Because the deployment of the Data Domain system is so simple, and additional copies of RMAN backups can be scripted directly into RMAN, a real-world test is relatively simple and non-disruptive. Networking How fast can the data move? Obviously, the faster the network and the greater the number of network paths, the faster data can move. However, there are other choke points in the communication path that must be considered. Many CPU s cannot pump data out on a network at full network bandwidths. Some network environments impose several switch hops between the RMAN server and the backup device. Finally, a given model of Data Domain system has a limit as to how many streams and how much data can be written into it at any given instant. These last numbers are available from any Data Domain system engineer, but currently range between MB/sec and 5-20 streams depending on model. As a useful datapoint, a single gigabit Ethernet can move about 125 MB/s, so to see more aggregate performance on a Data Domain system that is rated faster, more than one link and write stream would be needed simultaneously. Security Backup security to a Data Domain system is usually maintained by controlling access to the share points on the Data Domain system. By limiting access to specific directories on the Data Domain system, the DBA can ensure access control to the backup files. Further controls by setting UID/GID credentials (in a UNIX environment) or SID credentials (for Windows) can add additional control. Note: Full SID control in a Windows environment is not yet provided in a Data Domain system as of DDOS Version 4.3. But even with the existing limitations, access to the Oracle backup data can be restricted to only the Oracle machines themselves and the SID under which RMAN runs so the limitation should not be an issue. Best Practice Recommendation: It is recommended that the shares (both NFS and CIFS) accessed directly by Oracle RMAN be restricted to only the servers actually running Oracle RMAN and any servers that might be used for a potential recovery in the future. Optimizing Deduplication When dealing with any deduplication technology, a few key factors need to be kept in mind. Most importantly is the fact that multiplexing of backup datastreams is detrimental to deduplication. This is because all deduplication techniques rely on being able to quickly identify repeated data patterns. When backup data is multiplexed, there is the probability that the data presented to the backup device is in slightly different patterns each time. This will cause degradation in the deduplication performance. Multiplexing of backup data was introduced to address the performance issues affecting backup to tape. It was important that the backup process keep the tape drive fed with data so as to avoid shoe shining where the tape would have to stop and backup when it had to wait for more data to be presented. Backing up to a disk device does not suffer this problem so multiplexing has minimal value when disk is used. Important RMAN options Type of backup: Full, Incremental, Image Best Practice Recommendation: It is generally recognized that Oracle recovery processes work best against full backups. A full backup has all of the most recent information and is fastest to recover. Incremental backups were developed because some databases are just too large to make full backups every night. With deduplication technologies removing redundant copies of database segments, this is now less of an issue. However, a full backup to a Data Domain system does still involve moving all the data over a network connection. Some databases may simply be too large to send over a network in an acceptable period of time. If this is the case, incremental nightly backups are the only available solution with periodic full backups when the opportunity exists; usually weekends. If these are still not acceptable, Oracle presents other techniques which involve storing data on mirrored disks which can be separated and backups made from the mirror. Oracle also offers standby database solutions where the secondary database can be frozen and backups made from it. These advanced solutions are beyond the scope of this paper. However, a Data Domain system still makes an excellent backup repository for even these extremely large databases. FILESPERSET FILESPERSET is the option Oracle RMAN uses to control multiplexing of the data files into the backupsets. Best Practice Recommendation: Specify FILESPERSET = 1 or use image copy when backing up to the Data Domain deduplication storage system. As described earlier, FILESPERSET controls how many datafiles are written to a particular file within the backupset. Since the files involved are read in parallel and asynchronously, the blocks written to the backupset are likely to be in slightly different order each time. What does this mean when using a Data Domain deduplication storage system? While this does not affect the ability to recover the data from the backupset, it does reduce the ability for the Data Domain system to effectively de-duplicate the incoming data. Identical data to previous backups may come in but not be recognized as already-written because it is aligned slightly differently or is in a slightly different order. 10

11 Maxopenfiles Best Practice Recommendation: Specify MAXOPENFILES = 1 for each channel defined. This will ensure that each RMAN channel only reads from a single file at any one time. Setting FILESPERSET=1 should implicitly cause MAXOPENFILES to be one, but setting it explicitly guarantees there is no doubt. Parallelism (aka, Channels) Best Practice Recommendation: Define additional channels to increase the number of parallel backup processes running. With FILESPERSET=1 (specified per backup job) and MAXOPENFILES=1 (specified per channel) combining to limit the multiplexing of data, specifying additional channels and/or degrees of parallelism will allow RMAN to keep more data moving into the backupsets. The optimal number of channels will be determined by the available CPU and memory resources as well as I/O capacity. The rate at which a Data Domain system can receive data is dependent on model, and connectivity as well as the ability of the host OS to push data over the network. Encryption Best Practice Recommendation: Encryption of backupsets is supported by Oracle RMAN. However, like multiplexing, encryption changes the data patterns presented to the backup device. Unless the customer is certain that the encryption chosen will always encrypt the exact same data in exactly the same way each time, it is recommended that RMAN encryption NOT be used. Compression Best Practice Recommendation: As previously mentioned, RMAN provides two styles of compression. The unused block compression is the default for Oracle 10g and can only be disabled by doing an image (COPY) backup. Generally, most customers have been seeing acceptable deduplication when doing normal full backups. To invoke the BZIP2 compression of RMAN, the DBA specifies AS COMPRESSED on the BACKUP command. This style of compression is similar to the hardware compression facility provided by most modern tape backup facilities. Since this is done in software, it uses scarce CPU and memory resources during the backup that can be used for other purposes. Given that the Data Domain deduplication storage system implements a similar compression facility AFTER the incoming data has been deduplicated, it is more efficient to allow the system to perform the compression. Multiplexing Archivelog Destinations Best Practice Recommendation: Including the Data Domain system as a destination for log archiving provides two advantages. Primarily, a second copy of the archivelogs provides additional protection in the event that the primary copy is damaged or lost. Since the archivelogs will eventually be backed up to the Data Domain system as part of an RMAN full backup, this simply means that the files are getting to the Data Domain system sooner than when RMAN runs. The files are identical (even though RMAN renames them) as long as the FILESPERSET=1 guideline mentioned earlier is followed so the Data Domain system will detect the common data sequences and deduplicate the second copy so that little or no additional disk space is consumed on the Data Domain system. Additional protection without additional disk space consumption, something to consider taking advantage of. Marking this second destination to Oracle as OPTIONAL (as opposed to MANDATORY ) will allow Oracle to continue running in the event that the Data Domain system is unavailable for service or other duties. The changes required to achieve this setup are well documented in various Oracle administration manuals. Very Large Database Considerations Best Practice Recommendation: In the case where the database being backed up is consuming tens or hundreds of terabytes, it will be necessary to utilize multiple Data Domain systems to provide backup resources. This is easy to do, but some planning of the deployment is required. Keep in mind that the Data Domain system can only deduplicate data that it has seen before and has something to compare to. This implies that compression is maximized when the same data is presented to the Data Domain system repeatedly so it has a baseline and can store only the new data presented. If a large database, composed of possibly thousands of datafiles, is backed up to a series of Data Domain systems, RMAN (if left to its own defaults) will try to spread the backup load across the various targets. Since this is somewhat random and therefore non-deterministic, RMAN might well send some datafiles to Data Domain system 1 on the first backup but to Data Domain system 2 on a subsequent backup. Obviously this is not optimal. Remembering the earlier discussion of RMAN channels, we utilize channels to provide the necessary controls. In general, we would define one-or-more channels to each Data Domain system and then constrain each channel to backing up specific datafiles. So the Best Practice is to NOT specify AS COMPRESSED on the BACKUP command. 11

12 An example might look something like this: Questions and Answers Datafile 1 Datafile 2 Channel Allocation to Specific Datafiles Datafile 3 Datafile 4 Datafile 5 Archived redo logs 1. Turn on block checking. REASON: The aim is to detect, very early the presence of corrupt blocks in the database. This has a slight performance overhead, but Checksums allow Oracle to detect early corruption caused by underlying disk, storage system, or I/O system problems. channel ddr1 channel ddr2 channel ddr3 SQL> alter system set db_block_checking = true scope=both; Backupset 1 RUN { CONFIGURE CONTROLFILE AUTOBACKUP ON; ALLOCATE CHANNEL ddr1 DEVICE TYPE DISK FORMAT / ddmnt/ddr-a/backup/oraback ; ALLOCATE CHANNEL ddr2 DEVICE TYPE DISK FORMAT / ddmnt/ddr-b/backup/oraback ; ALLOCATE CHANNEL ddr3 DEVICE TYPE DISK FORMAT / ddmnt/ddr-c/backup/oraback ; BACKUP (DATAFILE 1,2,3 FILESPERSET 1 CHANNEL ddr1) (DATAFILE 4,5 FILESPERSET 1 CHANNEL ddr2) (ARCHIVELOG ALL FILESPERSET 1 CHANNEL ddr3 INCLUDE CURRENT CONTROLFILE); } This example also shows how the degree of parallelism can be controlled manually. This will run three simultaneous tasks backing up files to three different Data Domain systems. Oracle Recommendations For completion, we include a copy of an FAQ from Oracle s MetaLink website. We sincerely hope they will not mind: Subject: Top 10 Backup and Recovery best practices. Doc ID: Note: Type: FAQ Last Revision Date: 04-OCT-2007 Status: PUBLISHED Purpose Backupset 2 Backupset 3 Top 10 Backup and Recovery best practices. This document assumes that you are doing the Backup and Recovery basics. Running in Archivelog mode Multiplexing the controlfile Taking regular backups Backupset 4 Backupset 5 DDR-A DDR-B DDR-C Backupset 6 Periodically doing a complete restore to test your procedures. 2. Turn on block tracking when using RMAN backups (if running 10g) REASON: This will allow RMAN to backup only those blocks that have changed since the last full backup, which will reduce the time taken to back up, as less blocks will be backed up. SQL> alter database enable block change tracking using file /u01/oradata/ora1/change_tracking.f ; 3. Duplex log groups and members and have more than one archive log dest. REASON: If an archivelog is corrupted or lost, by having multiple copies in multiple locations, the other logs will still be available and could be used. If an online log is deleted or becomes corrupt, you will have another member that can be used to recover if required. SQL> alter system set log_archive_dest_2= location=/ new/location/archive2 scope=both; SQL> alter database add logfile member /new/location/ redo21.log to group 1; 4. when backing up the database use the check logical parameter REASON: This will cause RMAN to check for logical corruption within a block as well as the normal head/tail checksumming. This is the best way to ensure that you will get a good backup. RMAN> backup check logical database plus archivelog delete input; 5. Test your backup. REASON: This will do everything except actually restore the database. This is the best method to determine if your backup is good and usable before being in a situation where it is critical and issues exist. RMAN> restore validate database; 6. Have each datafile in a single backup piece REASON: When doing a partial restore RMAN must read through the entire piece to get the datafile/archivelog requested. The smaller the backup piece the quicker the restore can complete. This is especially relevent with tape backups of large databases or where the restore is only on individual / few files. RMAN> backup database FILESPERSET 1 plus archivelog delete input; 12

13 7. Maintain your RMAN catalog/controlfile REASON: Choose your retention policy carefully. Make sure that it compliments your tape subsystem retention policy, requirements for backup recovery strategy. If not using a catalog, ensure that your controlfile record keep time instance parameter matches your retention policy. SQL> alter system set control_file_record_keep_time=21 scope=both; This will keep 21 days of backup records. Run regular catalog maintenance. REASON: Delete obsolete will remove backups that are outside your retention policy. If obsolete backups are not deleted, the catalog will continue to grow until performance becomes an issue. See Note Managing multiple archive log destinations with RMAN for details. Summary A Data Domain System makes an excellent target for Oracle/RMAN backups because 1. A Data Domain system integrates easily and seamlessly into existing Oracle environments 2. Allows the DBA to retain online a greater number of full backup images thereby optimizing recovery options while occupying a minimal footprint in the datacenter 3. Greatly reduces the dependence on tape RMAN> delete obsolete; REASON: crosschecking will check that the catalog/controlfile matches the physical backups. If a backup is missing, it will set the piece to EXPIRED so when a restore is started, that it will not be eligible, and an earlier backup will be used. To remove the expired backups from the catalog/controlfile use the delete expired command. RMAN> crosscheck backup; RMAN> delete expired backup; 8. Prepare for loss of controlfiles. set autobackup on REASON: This will ensure that you always have an up to date controlfile available that has been taken at the end of the current backup not during. RMAN> configure controlfile autobackup on; keep your backup logs REASON: The backup log contains parameters for your tape access, locations on controlfile backups that can be utilised if complete loss occurs. 9. Test your recovery REASON: During a recovery situation this will let you know how the recovery will go without actually doing it, and can avoid having to restore source datafiles again. SQL> recover database test; 10. Do not specify delete all input when backing up archivelogs REASON: Delete all input will backup from one destination then delete both copies of the archivelog where as delete input will backup from one location and then delete what has been backed up. The next backup will back up those from location 2 as well as new logs from location 1, then delete all that are backed up. This means that you will have the archivelogs since the last backup available on disk in location 2 (as well as backed up once) and two copies backup up prior to the previous backup. 13

14 Appendix A Troubleshooting There have been situations where getting the FILESPERSET option set correctly has been a problem. The following displays are presented to assist in confirming the proper behavior of RMAN. Using Oracle Enterprise Manager If access to the Oracle Enterprise Manager (OEM) is available, the following screenshots provide the clue. In particular, we are looking for the backup report that shows how many datafiles have been placed into a backup set. If the setting is in effect, RMAN will create a backup set (one backup file) for each datafile and archived log in the database. Within OEM, navigate to the Maintenance -> Backup Reports screen. This screen should look something like this: As you can see, there are three backup jobs that completed recently. For this example, the third one (started at 2:07pm) will show a backup WITHOUT FILESPERSET=1, the second one (started at 2:17pm) will show a backup WITH FILESPERSET=1, and the newest one (started at 2:31pm) will show an Image (or Copy) backup with FILESPERSET=1. 14

15 Example: FILESPERSET not set to 1 Selecting the 2:07pm job shows: We have scrolled down a bit and only show the most important part of the display. Note that the report shows: Input Summary - Datafile: Files backed up: 5 - Archived Log: Files backed up: 1 - Control file: Files backed up: 1 - SPFILE: Files backed up: 1 Output Summary - Total Backup Sets: 3 Without FILESPERSET = 1, Oracle will combine multiple database files into a handful of backup files. In particular, all the datafiles can be combined into one-or-more backup files, all the archivelog files can similarly combined, and the controlfile can be combined with the parameter (SPFILE) file. When FILESPERSET is set to one, each datafile and archivelog are backed up to separate files. In this case, we see that RMAN only created 3 Backup Sets. What RMAN has done is combined all the Datafiles into a single backup set / file, all the archived logfiles (in this case only one) into a second backup set / file, and the Control file into a third. 15

16 Further down the screen we see: Note: Notice the column Compression Ratio. Even though we did not specify to RMAN to use compression, some compression is still done. Oracle describes this as unused block elimination which basically means that empty space within the Oracle data files is not sent to backup. This should not be confused with either the compression effect observed on the Data Domain nor compression that would be seen if RMAN performed its own optional compression. Toward the end of the report, we see the listing of the actual files RMAN created with their (rather obscure) filenames. These are the files actually written to the Data Domain system. So that ends our example of a backup performed without setting FILESPERSET and simply going with the Oracle default. 16

17 Example: FILESPERSET = 1 In this example, we specify FILESPERSET=1 for the backup steps for both the datafiles and the archivelogs. If the backup is configured with OEM, this can be set simply by filling in the field shown below and setting it to 1. This screen appears when performing a Customized backup using OEM. This results in a script being generated that looks like: 17

18 OK, if we run this script, how do the results vary. Below are the same three screen shots shown earlier with the first backup. 18

19 From the command line If Oracle Enterprise manager is not available, the required information can also be gained from the command line. The RMAN list backup command is used for this. An example is shown below. RMAN> list backup; List of Backup Sets =================== BS Key Type LV Size Device Type Elapsed Time Completion Time Full M DISK 00:01:42 01-NOV-07 BP Key: 2 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_ Piece Name: /dd580a/oracle/09ivv8cf_1_1 List of Datafiles in backup set 2 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/system01.dbf 2 Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/undotbs01.dbf 3 Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/sysaux01.dbf 4 Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/users01.dbf 5 Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/example01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time Full 6.80M DISK 00:00:05 01-NOV-07 BP Key: 3 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_ Piece Name: /dd580a/oracle/0aivv8fo_1_1 Control File Included: Ckp SCN: Ckp time: 01-NOV-07 SPFILE Included: Modification time: 01-NOV-07 BS Key Size Device Type Elapsed Time Completion Time M DISK 00:00:12 01-NOV-07 BP Key: 4 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_ Piece Name: /dd580a/oracle/0bivv8gb_1_1 Thrd Seq Low SCN Low Time Next SCN Next Time NOV NOV-07 BS Key Type LV Size Device Type Elapsed Time Completion Time Full M DISK 00:01:42 01-NOV-07 BP Key: 5 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0civv90n_1_1 List of Datafiles in backup set 5 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/system01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time Full 26.17M DISK 00:00:05 01-NOV-07 BP Key: 6 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0divv940_1_1 List of Datafiles in backup set 6 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/undotbs01.dbf 19

20 BS Key Type LV Size Device Type Elapsed Time Completion Time Full M DISK 00:00:38 01-NOV-07 BP Key: 7 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0eivv947_1_1 List of Datafiles in backup set 7 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/sysaux01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time Full 56.38M DISK 00:00:13 01-NOV-07 BP Key: 8 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0fivv95k_1_1 List of Datafiles in backup set 8 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/example01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time Full 1.90M DISK 00:00:01 01-NOV-07 BP Key: 9 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0givv963_1_1 List of Datafiles in backup set 9 File LV Type Ckp SCN Ckp Time Name Full NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/users01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time Full 6.77M DISK 00:00:02 01-NOV-07 BP Key: 10 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0hivv964_1_1 Control File Included: Ckp SCN: Ckp time: 01-NOV-07 BS Key Type LV Size Device Type Elapsed Time Completion Time Full 80.00K DISK 00:00:01 01-NOV-07 BP Key: 11 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0iivv966_1_1 SPFILE Included: Modification time: 01-NOV-07 BS Key Size Device Type Elapsed Time Completion Time M DISK 00:00:02 01-NOV-07 BP Key: 12 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_ Piece Name: /dd580a/oracle/0jivv96a_1_1 List of Archived Logs in backup set 12 Thrd Seq Low SCN Low Time Next SCN Next Time NOV NOV-07 BS Key Type LV Size Device Type Elapsed Time Completion Time Full 80.00K DISK 00:00:01 01-NOV-07 BP Key: 13 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000008_ Piece Name: /dd580a/oracle/0qivva1h_1_1 SPFILE Included: Modification time: 01-NOV-07 20