VERITAS Volume Replication. and Oracle Databases

Transcription

1 A SOLUTIONS WHITE PAPER from VERITAS Software Corporation VERITAS Volume Replication and Oracle Databases VERITAS volume replication technology, and how it enhances disaster tolerance in Oracle databases

2

3 Abstract Remote replication of online disks and volumes is emerging as the technique of choice for protecting enterprise data against environmental disasters. Recognizing this, VERITAS has embarked on aggressive campaign to build disaster recovery solutions around the VERITAS Volume Replicator for major UNIX platforms. Oracle Corporation, the leading supplier of relational database technology for the enterprise, recognizes the importance of disaster recoverability as well, and has introduced its own replication technology that works cooperatively with the volume replication technologies of other vendors such as VERITAS. In addition, Oracle has instituted an audited testing program to verify that its technology interoperates correctly with other vendors volume replication technologies for database disaster recovery. VERITAS is proud to be the first vendor of volume replication technology to successfully complete the Oracle Storage Compatibility Program (OSCP) suite of disaster recoverability tests. This paper describes the capabilities of the VERITAS Volume Replicator, and how it interoperates with Oracle s standby database replication technology to provide robust disaster recoverability for enterprise Oracle databases. 3

4

5 Contents I. ONLINE STORAGE FOR THE ENTERPRISE... 7 ENTERPRISE DATA STORAGE REQUIREMENTS... 7 THE SOLUTION: RELATIONAL DATABASES AND LOGICAL VOLUMES... 7 THE VERITAS VOLUME MANAGER... 8 II. DATA REPLICATION THE NATURE OF DATA REPLICATION THE VERITAS VOLUME REPLICATOR SYNCHRONOUS AND ASYNCHRONOUS REPLICATION III. VOLUME REPLICATION AND ORACLE ORACLE DATABASE REPLICATION TECHNIQUES HYBRID REPLICATION OF ORACLE DATABASES COMPARISON OF ORACLE REPLICATION TECHNIQUES IV. SUMMING UP VERITAS TOTAL DATA STORAGE MANAGEMENT WHY THE VERITAS SOLUTION? Figures FIGURE 1: REPLICATION WITH VERITAS VOLUME REPLICATOR FIGURE 2: VERITAS VOLUME REPLICATOR DATA FLOW FIGURE 3: SYNCHRONOUS REPLICATION TIME LINE FIGURE 4: APPLICATION PERFORMANCE WITH SYNCHRONOUS AND ASYNCHRONOUS REPLICATION FIGURE 5: IN-BAND CONTROL MESSAGE TIME LINE FIGURE 6: RECIPROCAL DISASTER RECOVERY Tables TABLE 1: VOLUME TYPES SUPPORTED BY THE VERITAS VOLUME MANAGER... 8 TABLE 2: COMPARISON OF STANDBY DATABASE HYBRID AND PURE VOLUME REPLICATION

6

7 I. Online Storage for the Enterprise High-performance continuous access to online data Enterprise Data Storage Requirements The Requirement: Continuously Accessible, Trustworthy Data Continuous access to data has become an operational necessity for enterprises. Customers, suppliers, and even employees expect to be able to access enterprise information resources all the time. Downtime, whether due to hardware or software failure, the need to back data up, or because an upgrade is necessary, is no longer an option. For managers and administrators responsible for information services data storage management strategies, the challenges of continuous data availability are formidable: Maintaining the integrity of enterprise data while meeting increasingly complex, rapidly changing application needs. Providing continuous access to data, even if applications, systems, or entire data centers become incapacitated. Providing I/O performance that scales with enterprise requirements. Containing management cost as the amount of data increases. The Solution: Relational Databases and Logical Volumes Technology offers two complementary answers to enterprise needs for correct, continuously available, scalable, manageable data: relational database management systems and logical storage volumes. For data integrity, relational database management systems such as Oracle provide: enterprise-wide application-independent data formats, data item integrity through constraints and user-definable triggered procedures, transactional integrity through logging and commitment/rollback techniques, high I/O performance, through intelligent use of massive cache memories, manageability through extensive management toolsets. Relational database management systems do an outstanding job of keeping data errorfree and accessible at high performance levels. To do their job optimally, however, 7

8 they need storage they can rely on. Oracle is more robust, performs better, and is more economical to own in the long run if the files or raw storage that hold its user data and metadata are likewise robust, high performing, and flexibly manageable. Logical volume management technology aggregates disks, and presents the database management system with online storage units that are functionally disk-like, but have superior availability, performance, and manageability. Software packages such as the VERITAS Volume Manager provide logical volume management. RAID controllers also provide a level of logical volume management. Wherever it runs, volume management software has two basic responsibilities: It protects data against loss or loss of access due to disk or channel hardware failures. Mirroring and RAID are the two techniques commonly employed. It maps disk-like volume block address spaces to disk blocks for optimal data access performance. Data striping is the most common mapping technique. The VERITAS Volume Manager The VERITAS Volume Manager runs as an operating system layer between I/O drivers and the file system or database management system, VERITAS Volume Manager enhances storage availability, performance, and manageability by presenting parts of disks, entire disks, or groups of disks as logical volumes. Table 1 indicates the types of volumes supported by the VERITAS Volume Manager in Solaris, HP-UX, and Windows server operating system environments. Volume Type Mirrored (two or more copies) ( RAID 1 ) Striped and Mirrored (n-way) Striped with Parity ( RAID 5 ) Striped without Parity ( RAID 0 ) Concatenated (capacity aggregated) Disk Failure Tolerance (Data Availability) Very high Very high High Lower than single disk Lower than single disk Performance (relative to single disk) High read performance Comparable write performance Very high read performance High write performance High read performance Low write performance High read and write performance Comparable read and write performance Enterprise IT Application Small (single-disk) critical files Large (multi-disk) critical files Important read-mostly data Limited to easily replaceable performance critical data Limited to easily replaceable nonperformance critical data Simple Same as single disk Same as single disk Small, easily replaceable non-performance critical data Table 1: Volume Types Supported by the VERITAS Volume Manager 8

9 RAID array control software exploits specialized hardware for best RAID array performance. Host-based volume managers tend to offer easier manageability because they run in richer software environments that are more closely integrated with operating systems, file systems, and applications. For example, VERITAS Volume Manager manageability features include: Three (or more) way mirroring. This is useful when a frozen image of operational data is required for backup or application testing. With three mirrored copies, one can be used for backup or testing while live applications continue to run with failure tolerant data. Snapshots, or point-in-time frozen images of volume data for ad hoc backup, query, testing, or reporting purposes. Online volume expansion coupled with file system expansion, enabling an administrator to handle unplanned storage needs without application disruption. Heterogeneous device support. VERITAS Volume Manager aggregates (mirror, stripe, concatenate or RAID) both physical disks of different types and the virtual disks presented by RAID array controllers. Volume Manager Distance Limitations Volume managers provide the robust, high-performance storage substrate that file systems and databases need. They keep data available if disks, channels, or in cluster environments, servers fail as long as everything is in one computer room. RAID arrays and volume managers are designed around highly reliable local channels that communicate with all disks at about the same speed. When writing to a mirrored volume, for example, a volume manager typically issues simultaneous write commands to all disks. Application response and error recovery algorithms both assume that all commands to devices complete in roughly the same amount of time. Writing to a RAID volume requires more commands, but again, algorithms assume that no one command takes markedly longer than any other to execute. Similarly, all disks and channels are assumed to be equally reliable. When a disk command fails, the volume manager retries it. If the retries fail, the volume manager adopts permanent failure mode algorithms for the volume. For example, when one disk of a mirrored volume fails, the volume manager transparently redirects all I/O requests to the volume s other disk(s). The assumption of reliable channels and equidistant devices is valid for computer rooms where interconnects are short and reliable. In terms of response time, all storage devices are equidistant from all host computers. Business needs, however, have 9

10 created requirements for maintaining identical copies of data in situations where channels are not reliable and where response time is not the same for all devices. Enterprises need to recover quickly from fire, flood, vandalism, power grid failure, software failure, and other events that can incapacitate entire data centers. The need to recover from disasters has led organizations to establish widely separated recovery data centers. Recovery data centers are outside an expected disaster radius, and contain sufficient equipment to run critical applications if the primary data center is incapacitated. For a disaster recovery site to be effective, it must have current data for critical applications. It is often useful to reuse operational and historical data without affecting online applications, for example to test new applications or to discover trends by mining data. Applications that reuse data typically require dedicated computers and storage, but they also need access to recent operational data. Distributed operations sometimes require that data be published from a central creation point to multiple distributed usage sites. Price lists, product specifications, and web pages all fall into this category. These data must be identical throughout an enterprise. These applications all require that data be copied between systems while it is in use, often over links that are less reliable and have lower performance than I/O channels. If conventional mirroring were to be used for these purposes: performance of the enterprise s online applications would be throttled by the I/O time to write data to the remote location, and, online application availability would be reduced, since there would necessarily be an application outage each time the link between local and remote data failed momentarily. Data Replication Technology Data replication technology overcomes these limitations and meets business needs by mirroring data over long distances and unreliable links. Unlike mirroring, replication technology assumes that: writing data to remote devices can take significantly longer than writing to local devices, and, remote links are less reliable than local channels, and in particular, are subject to momentary recoverable outages. Data replication can be configured to eliminate remote write time from application response time by allowing I/O to be buffered locally for later delivery in case of momentary network overload or outage. Alternatively, a system administrator can con- 10

11 figure replication to maintain strict synchronization between local and remote sites. The following sections describe data replication as implemented by the VERITAS Volume Replicator, and how the VERITAS Volume Replicator can be used to enhance the disaster recoverability and utility of Oracle databases. 11

12

13 II. Data Replication Protecting data against disaster The Nature of Data Replication Whether the purpose is reuse or disaster recovery, the mechanism of data replication is the same. A primary replication manager intercepts live data updates at the data processing site, and distributes them to one or more secondary replication managers at remote sites, where they are stored persistently. Secondary replication managers maintain a defined level of synchronization with the primary replication manager. Replication is designed for links that are too slow to permit mirroring, or for which transient overloads and failures are expected. Primary replication managers can queue data updates for delayed delivery to secondary sites if necessary. The VERITAS Volume Replicator The VERITAS Volume Replicator, illustrated in Figure 1, is an adjunct to the VERITAS Volume Manager. The VERITAS Volume Replicator keeps the contents of up to 32 secondary volume groups synchronized with the contents of a primary volume group that holds live data being processed by applications. The VERITAS Volume Replicator maintains the integrity of replicated volumes in case of network or system outages without significantly degrading application performance. r Application Application Updates Updates Application Write Primary Server P Oracle RDBMS Storage Replicator Volume Manager Update Update Update Volume V File System /fs File A.DAT File B.DAT Volume W File System /oracle File T1.ORA File T2.ORA Volume Group G Secondary Server S1 Storage Replicator Volume Manager Update Update Update Volume V1 File System /fs File A.DAT File B.DAT Volume W1 File System /oracle File File Update Update Update SRVM propagates application updates to volume groups at secondary sites Volume Group G 1 Volumes may be simple, concatenated, striped or mirrored Secondary Server S2 Storage Replicator Volume Manager Volume Group G2 Volume V2 File System /fs File A.DAT File B.DAT Volume W2 File System /oracle File File Figure 1: Replication with VERITAS Volume Replicator 13

14 In Figure 1, applications run on the primary server, accessing data on volumes in Volume Group G. Applications may access data directly through a file system (e.g., /fs), or through a database management system that uses either files or raw volumes (e.g., /oracle). In this example, VERITAS Volume Replicator replicates updates from Volume Group G to volume groups G 1 and G 2 on secondary servers S 1 and S 2. Figure 1 illustrates four key points about VERITAS Volume Replicator: Volume replication mirrors block updates to volume contents, whether the volumes blocks contain files used directly by applications, file systems containing databases, or raw storage used directly by a database management system. Volume replication has a single source (the primary site) and one or more targets (secondary sites). Applications run at the primary site. Secondary replicated volumes cannot be accessed by applications while replication is active. Data are replicated over an arbitrary network. Replication does not require special hardware or dedicated network connections (although VERITAS recommends dedicated network links in most instances for predictable response). In principle, data can be replicated across transcontinental distances. Distance limits are a function of application performance needs and affordability. Although VERITAS Volume Replicator maintains identical contents, secondary volumes need not be of the same type as primary volumes. For example, a 3-way mirrored volume at the primary site might be replicated to 2-way mirrored volumes, or even to non-failure tolerant simple secondary volumes. Overview of Volume Replication with VERITAS Volume Replicator VERITAS Volume Replicator replicates each update to a primary volume on all corresponding secondary volumes, regardless of the source of the request. VERITAS Volume Replicator preserves update ordering to ensure that secondary volume contents are in a correct state at all points during a sequence of updates. VERITAS Volume Replicator is unaware of I/O request interdependencies. It is impossible for a file system or database management system at a secondary site to determine whether data or metadata updates are in progress. This is why secondary replicated volumes cannot be used by applications during replication. The after-the-fact usage characteristic of volume replication makes it most suitable for two general application categories: data reuse: Application testing and data mining both require access to operational and recent historical data. Obviously, testing cannot be done with operational data. Data mining typically requires dedicated computing and I/O resources, so it, too, needs its own copy of operational data. 14

15 disaster recovery: A disaster recovery site is a secondary data center located far enough from the primary data center that it can continue operating if a disaster occurs. Primary site volume groups containing operational data can be replicated to a disaster recovery site. If a primary site disaster occurs, applications can be restarted at the disaster recovery site, and processing can resume using the (current) replicated data. Both of these application classes have the characteristic that data at the secondary site is used by applications after replication stops, rather than while replication is occurring. 1 VERITAS Volume Replicator and the VERITAS Foundation Since VERITAS Volume Replicator replicates volume contents, VERITAS Foundation Suite features are available for replicated volumes at both primary and secondary sites, including: availability enhancements such as mirroring, general-purpose performance enhancements such as data striping, and, database-specific performance enhancements such as Quick I/O for Databases. Policy-Based Management VERITAS Volume Replicator is policy-based. System administrators set replication policies to meet business requirements. Replication policies include: which volumes at the primary site are to be replicated, which volumes at secondary sites are to be replication targets, the degree of synchronization between primary and secondary sites, and, how temporary failures such as network outages are handled. Once policies have been set, replication is automatic, requiring no system administrator intervention until an exceptional event such as a site disaster actually occurs. VERITAS Volume Replicator and the VERITAS Volume Manager From a technical standpoint, VERITAS Volume Replicator is closely integrated with the VERITAS Volume Manager. The executable images for both products are distributed together, although they are enabled by separate license keys. VERITAS Volume Replicator requires that VERITAS Volume Manager be installed and enabled. 1 With VERITAS Volume Replicator, secondary replicated volumes can be mirrored. Mirrors can be broken off at instants when primary site applications are known to be quiescent, and mounted for application use at the secondary site. In this case, the broken mirror of replicated data is used after replication stops, but replication of the operational copy of data resumes as soon as the mirror has been broken off and applications are re-enabled. 15

16 Synchronous and Asynchronous Replication VERITAS Volume Replicator at the primary site logs all write requests to replicated volumes in a Storage Replicator Log (SRL) before sending them to secondary sites. The SRL is VERITAS Volume Replicator s mechanism for riding through transitory network outages and overloads. Application Updates Application Application Updates Updates Application Updates Primary Server Oracle RDBMS Storage Replicator Volume Manager Volume V File System /fs File A.DAT File B.DAT Volume W File System /oracle File T1.ORA File T2.ORA Volume L p Primary SRL Volume V File System /fs File A.DAT File B.DAT Volume W File System /oracle File T1.ORA File T2.ORA Volume L s Secondary SRL Secondary Server Storage Replicator Volume Manager Figure 2: VERITAS Volume Replicator Data Flow If a network link fails, VERITAS Volume Replicator at the primary site continues to log updates to replicated volumes. After the network recovers, logged updates are sent to affected secondary sites. Secondary sites with functioning network links are updated as write requests occur at the primary site. The behavior of VERITAS Volume Replicator during a network outage is determined by whether the system administrator has set the volume group to replicate synchronously or asynchronously. Synchronous Replication Replicated data at a secondary site is current if volume group contents are identical to their primary site counterparts. If data at a secondary site is to always be current, replication must be synchronous. Each update must be logged persistently at the primary site and received by all secondary sites before it is considered complete. When replicating synchronously, VERITAS Volume Replicator maintains primary and secondary site data synchronization. An application write request to a synchronously replicated volume is considered complete as soon as the update is: logged at the primary site, and, transmitted to and acknowledged by all secondary sites. 16

17 Secondary sites confirm updates in two stages. The first confirmation acknowledges receipt of the update. The second, indicating that the primary site need no longer keep the update in its SRL, is sent when data is on disk at the secondary site. Figure 3 is a time line for synchronous replication, showing actions that can occur concurrently. An update to a synchronously replicated volume takes longer than an update to a non-replicated volume by a round trip message time to the most distant (in time) secondary site. Write data to primary site disk Confirm receipt to primary site Signal completion to application Application may proceed when data is written to primary disk and received by secondary site Analyze request Write data to primary log Send data to secondary site These can be concurrent write to disk at primary site and transmission to secondary site may be concurrent Event durations are not shown to scale Write data to disk at secondary site Confirm write to primary site Time Application need not wait for write to disk at secondary sites Figure 3: Synchronous Replication Time Line With the replication algorithm illustrated in Figure 3, data from write requests for which completion has been signaled are secure against: primary site disaster, because a copy exists at each secondary site, and, secondary site or communications link failures, because all updates are logged at the primary site. Synchronous Replication and Link Failure With synchronous replication, an application write request is not complete until the volume updates it implies have been acknowledged by secondary sites. This means that application writes cannot complete if network link fails. VERITAS Volume Replicator allows a system administrator to set one of three link outage policies for each replicated volume group: signal write failure to applications, abandon replication, reverting to local-only operation, or, switch to asynchronous replication (described below). Signaling write failure to applications effectively transfers responsibility for dealing with the outage to them. Abandoning replication entirely requires that a new base line copy of data be created at affected secondary sites before replication can recommence. Switching to asynchronous replication allows applications to continue operat- 17

18 ing, and accumulates updates locally for later transmission to secondary sites. In practice, most VERITAS Volume Replicator installations set up to switch to asynchronous replication when a link or secondary site failure occurs. Asynchronous Replication Application I/O peaks, network overloads, or simply a large number of secondary sites can elongate I/O response from synchronously replicated volume undesirably. For these situations VERITAS Volume Replicator supports asynchronous replication. When replication is asynchronous, an application write completes as soon as VERITAS Volume Replicator has logged the udpate in the volume group s SRL. Transmission and writing to secondary volumes is concurrent with continued application execution. Figure 4 illustrates the timing differences between synchronous and asynchronous replication. Synchronous replication Analyze request Write data to primary site log Write data to primary site disk Send data to secondary site Confirm receipt to primary site Signal completion to application These can be concurrent Application may proceed when data has been written to primary disk and received by secondary it Asynchronous replication Analyze request Write data to primary site log Write data to primary site disk Send data to secondary site Primary site signals completion to application Write data to disk at secondary site Confirm write to Application need primary site not wait for write to disk at secondary sites Time Event durations are not shown to scale Asynchronous replication reduces application response time by this amount Confirm receipt to primary site Write data to log at secondary site Write data to disk at secondary site Application may proceed when data has been written to log at primary site None of these affect application response Confirm write to primary site Figure 4: Application Performance with Synchronous and Asynchronous Replication As Figure 4 illustrates, asynchronous replication reduces application write response time. The more important consequence of asynchronous replication, however, is that it enables applications to continue to execute without performance degradation during momentary network outages and overloads. Asynchronous Replication and Network Loading VERITAS Volume Replicator sends asynchronous updates to secondary sites as quickly as network and secondary site loading permit. If a network overload is transient, updates queued at the primary SRL are eventually sent, and secondary site data becomes current. If the network overload is chronic, the number of unsent writes grows, and eventually, the SRL fills. When this occurs, VERITAS Volume Replicator implements one of several policies described below. Asynchronous replication pro- 18

19 tects against short-term network overload, but it is not a substitute for inadequate steady state network bandwidth. Limiting Exposure with Asynchronous Replication The advantages of asynchronous replication are: better application response compared to synchronous replication (as Figure 4 illustrates), application ride-through of momentary network outages and overloads, and, recoverability from secondary site crashes. The disadvantage is that there can be (usually) brief periods of time during which data on secondary volumes are not completely current. If a secondary computer crashes or a communication link fails with updates logged at the primary site but not yet transmitted to one or more secondary sites, secondary site data may not be current. Updates in the primary site SRL are transmitted to the secondary site and written after recovery. If the primary site suffers an unrecoverable disaster, and its SRL is destroyed, secondary site recovery may start with consistent but not-quite-current data. To limit this exposure, a system administrator can limit the number of updates by which secondary sites are permitted to be out of date. When the limit is exceeded, application write requests stall (no completion signal is given) until the number of unsent updates has fallen below a permissible threshold. Although it does not guarantee absolute data currency, asynchronous replication is often required for performance reasons. With synchronous replication, momentary write overload or network load from other sources can increase response times unacceptably. Asynchronous replication removes network dependencies from the application response path, and may make replication practical where it might otherwise not be. Storage Replication Log Overflow VERITAS Volume Replicator s Storage Replication Log is stored on a dedicated volume (which should be mirrored for failure tolerance). For maximum efficiency, the SRL is structured as a circular buffer that cannot be expanded during replication. If a lengthy network outage or long period of heavy update activity causes the SRL to fill, VERITAS Volume Replicator switches to keeping track locally of which regions of replicated volumes are modified. When bandwidth is again available, VERITAS Volume Replicator sends the contents of modified regions to affected secondary sites. When primary and secondary sites are again synchronized, replication resumes. Other VERITAS Volume Replicator policies for dealing with SRL overflow include: 19

20 abandoning replication: Since it requires full resynchronization before replication can recommence, this policy is seldom used in practice. delaying I/O request completion: VERITAS Volume Replicator will delay completion of application I/O requests until logged updates have been transmitted and a predetermined amount of SRL space is freed. An override causes replication to be abandoned if the SRL fills while a network link is down. This circumvents application failures that might result from repeated replication-related I/O request timeouts. Getting Replication Started Replication requires a starting point at which the contents of primary and secondary volumes are known to be identical. Thus far, the discussion has assumed that there is a starting point at which the contents of corresponding primary and secondary volume groups are identical, without discussing how they might have become identical. A replication starting point can be established by making an image (block-by-block) backup of primary volumes (e.g., using the UNIX dd utility for small volumes, or an image backup utility such as VERITAS NetBackup s for larger ones). When the image has been restored at each secondary site, replication can begin. This procedure may not be workable if it is impractical to freeze primary site data until all secondary site restores are complete. To minimize the impact of backup-based startup, VERITAS Volume Replicator includes a replication checkpoint facility that enables volumes to be used while preparing for replication. Immediately prior to starting a primary volume image backup, an administrator declares a replication checkpoint, causing a marker to be placed in the SRL. When the backup is complete, the administrator places a checkend marker in the SRL. Any updates to primary volumes between the checkpoint and checkend markers are reflected in the SRL, bracketed by the checkstart and checkend markers. The primary volumes may be used by applications during the backup. The completed image backup is then restored at secondary sites, and logical links with the primary site are established. VERITAS Volume Replicator sends all updates starting at the checkpoint marker to the secondary sites. When the checkend marker is reached, primary and secondary volume contents are identical, and synchronous or asynchronous replication, including update ordering, begins. VERITAS Volume Replicator auto-synchronization can simplify replication startup even further in some cases. To auto-synchronize, a system administrator establishes a replication link between primary and secondary volume groups, and VERITAS Vol- 20

21 ume Replicator copies the entire primary volume group contents to the secondary volumes. To expedite the copy, write ordering is not preserved during autosynchronization. Applications may update volumes during auto-synchronization. VERITAS Volume Replicator tracks primary volume updates by block region. When the copy is complete, updated regions are re-copied. When none of their block regions differ, primary and secondary volume groups are synchronized, and normal replication begins. VERITAS Volume Replicator In-Band Control VERITAS Volume Replicator maintains application write order between primary and secondary sites to ensure that newer application updates are not overwritten by older updates of the same blocks that arrive out of order because of network delays. This is necessary, but not sufficient to guarantee the consistency of volume contents from a file system or database management system viewpoint. When a file system or database management system has: no transactions outstanding, and, no updated data in cache that has not yet been written to disk, its on-disk image is said to be internally consistent. Internally consistent file system and database images greatly simplify backup and disaster recovery. Because VERITAS Volume Replicator deals with volumes, it cannot determine when a database or file system s on-disk image is internally consistent. Special in-band control application programming interfaces (APIs), however, allow registered applications at a primary site to send messages to registered applications at secondary sites within a replicated data stream. An in-band control message suspends updates at the receiving secondary site until a registered application has processed it and used a companion API to resume replication. Figure 5 illustrates in-band replication control. Write data-1 to replication log Send in band control message Write data-2 to replication log Write data-3 to replication log Write data-4 to replication log etc. Primary site continues to update replicated data Primary site Write data-1 to primary disk Write data-2 to primary disk Write data-3 to primary disk Write data-4 to primary disk Send data-1 to secondary site Send data-2 to secondary site Time Send data-3 to secondary site Secondary site Receive in band control message Write data-1 to secondary disk Process in band control message Updates are suspended when in band control message is received Invoke resume updates API Secondary site application uses API resume updating Write data-2 to secondary disk etc. Figure 5: In-Band Control Message Time Line 21

22 A primary site application can send an in-band control message when some significant event occurs (e.g., close of a business day). The receiving secondary site can respond to the event (e.g., by creating a VERITAS Volume Manager mirror frozen image of the day s data), and then allow secondary site updates to resume. Reciprocal Disaster Recovery The primary and secondary site roles are specific to each replicated volume group. A server may be the primary site for one volume group and a secondary site for another. Figure 6 illustrates the use of this feature for reciprocal disaster recoverability. Application A Application pdates A Application A Updates Application Updates A Updates Primary Server for Application A Application A Volumes File A!.ORA Volume File A2.ORA Volume Group A Volume Group B Application B Volumes File B1.ORA Volume File B2.ORA Primary Server for Application B Application B Application pdates B Application B Updates Application BBUpdates B Updates Secondary Server for Application B Application B Replica Volume Replication direction Application A Replica Volume Secondary Server for Application A Volume Group B Volume Group A Figure 6: Reciprocal Disaster Recovery The system illustrated in Figure 6 has servers dedicated to applications A and B. Data for application A are stored on Volume Group A, which is replicated to Volume Group A on application B s server, and conversely. If either site is incapacitated, its application can be restarted on the server at the surviving site with current data. For enterprises with large numbers of servers, reciprocal disaster recoverability based on replication technology can dramatically improve overall IT resiliency. For a hardware investment of: the storage devices required for replicated data, incremental server processing power and memory capacity to handle replication and provide adequate performance in the event of fail over, and, sufficient network bandwidth to accommodate replication traffic in addition to operational traffic, an enterprise can position itself to rapidly resume operations when a disaster incapacitates any of its critical applications. 22

23 III. Volume Replication And Oracle Disaster protection for Oracle databases Oracle Database Replication Techniques By one analyst s estimate, relational database management systems manage up to 90% of the online business information stored on servers, with Oracle holding the largest market share. As enterprises become increasingly conscious of the importance of their operational data and begin to plan for disaster recoverability, they naturally consider replication of their Oracle databases. Aware of this need, the Oracle Corporation has begun to offer a comprehensive testing-based certification program by which vendors of other replication products can verify that their products properly recover Oracle databases from site disasters, either by themselves or in conjunction with Oracle s own Replication Manager. This section describes techniques for using VERITAS Volume Replicator to replicate Oracle databases both with and without assistance from the Oracle Replication Manager. Volume Replication of Oracle Databases VERITAS Volume Replicator can replicate a complete Oracle database. Any of the techniques described earlier can be used to establish a replication starting point with identical database volume images at primary and secondary sites. Autosynchronization (page 21) uses the network to establish the starting point. For very large databases, a full database backup coupled with the replication checkpoint technique described on page 21 may be more practical. In either case, the database may be used during initial synchronization, but VERITAS Volume Replicator does not order writes until initial synchronization is complete. For disaster recovery, replicated database volumes must contain database control files and redo logs as well as data files. Synchronous replication is clearly preferable to preserve committed transactions, but performance considerations may require asynchronous replication for long distances or low performance network links. From a database standpoint, an unrecoverable disaster at the primary site is approximately equivalent to a local system crash at the secondary site. To activate the secondary site database for application use: replication is stopped, 23

24 secondary site replicated volumes are changed from secondary mode to primary mode and mounted, and, Oracle s crash recovery procedures are used to restore the database image to the most current transactionally consistent point possible by applying all complete transactions from the database redo log and undoing any incomplete ones. Oracle Standby Database Replication 2 Another option for making Oracle databases recoverable is Oracle s built-in replication facility based on database transactions rather than volume block updates. As Oracle processes transactions, it captures them in an online redo log that can be used to rebuild a current database from a backup if necessary due to hardware, software, or operations failure. Periodically, the online redo log is emptied into an archive redo log, and a new online redo log is started. Oracle s built-in replication facility uses these archive redo logs. Standby database replication starts with the creation of a complete point-in-time copy of an inactive database at a remote site. When the remote database, called a standby database, is identical to the primary database, replication begins. Each time an online redo log is closed and copied to an archive redo log, Oracle s Replication Manager transmits the archive redo log to the remote site and applies the transactions in it to the standby database. If a disaster incapacitates the primary site, the standby database is or can be made as current as the time of the newest archive redo log transaction. The advantage of standby database replication is that the database is always recovered to transactional consistency. In other words, the recovered database may be out of date (by the age of the newest archive redo log), but it does not contain partial transactions (e.g., debits without matching credits), or metadata updates, and can therefore be started and used immediately, without crash recovery. Hybrid Replication of Oracle Databases The disadvantage of standby database replication is that it does not restore a database to the most current state reflected in the online redo log. Standby database replication can be combined with volume replication of redo logs to improve the currency of a recovered database. 2 Oracle database replication can also be used to distribute a database, allowing read/write access to data at multiple sites from multiple sites, with user-defined policies for resolving conflicts. Since this usage is not comparable to volume replication, it is not discussed in this paper. 24

25 The most recent transactions against an Oracle database are recorded in the database s online redo log. The file containing this log must always be open, because the database management system is continually adding to it. For performance reasons, the must also be located near the database (i.e., at the primary site). More up to date standby database recovery would be possible if the Oracle online redo log could somehow be instantly transported from the disaster site to the recovery site. Cooperation between Oracle Replication and VERITAS Volume Replicator But transporting data instantly and transparently from a primary site to disaster recovery sites is exactly what VERITAS Volume Replicator does. Allocating the Oracle online redo log in a replicated volume makes the log available at the disaster recovery site. In a disaster recovery scenario, after all archive redo logs have been applied to a standby database, the replicated online redo log can be applied to roll the database forward to the point of the last complete primary site transaction. The Oracle Storage Compatibility Program Up to date database recovery after a site disaster is obviously extraordinarily valuable to enterprises. Recognizing this, Oracle Corporation has instituted an Oracle Storage Compatibility Program (OSCP), an audited program of pre-defined tests that verify the ability of a data replication solution to support up to date transactionally consistent recovery of Oracle databases at remote sites. Any storage configuration that supports replication of Oracle databases is eligible to participate in OSCP, including both hardware solutions that replicate data between disk controllers, and hybrid solutions such as VERITAS Volume Replicator, that combine basic storage hardware with host-based software. VERITAS has submitted the hybrid VERITAS Volume Replicator/Standby Database Replication technique described above to the OSCP testing process, and is the first vendor to successfully pass Oracle s tests. VERITAS Volume Replicator is approved by Oracle Corporation for up to date disaster recovery of Oracle databases. Comparison of Oracle Replication Techniques Whether standby database replication augmented by the volume replication of the Oracle online redo log, or full replication of the volumes holding the database and other application data is preferable depends on enterprise priorities. For example, if a database must be replicated across thousands of miles, data propagation time might preclude synchronous volume replication because of the adverse effect on application responsiveness. Similarly, if database corruption by unstable applications is a con- 25

26 cern, the standby database technique with delayed application of archive redo logs would probably be the preferred solution. On the other hand, if rapid recovery from a primary site disaster is the main concern, volume replication is probably the best solution, because the recovery process is less time consuming and complex than with standby database recovery. Table 2 compares seven key aspects of the hybrid and full volume replication techniques. Characteristic Standby Database Replication with Volume Replication for Online Redo Log Cost Potentially lower communication cost because archive redo logs are batched (greater efficiency) and are not latency-critical Replication Granularity Individual tables Protection against Database Corruption Database Operational Performance Failover Time (time until applications can begin using database at secondary site) Failback Time (move operational database back to original primary site) Better, because application, operating system, or storage subsystem-caused corruption is not reflected in the standby database until archive redo logs are applied. No impact on database I/O; minor impact possible when archive redo logs are copied and sent to secondary site; some impact if replication of the online redo log is synchronous. Possibly longer. All archive redo logs not yet applied to the standby database must be applied, and the online redo log must be replayed. Typically much shorter if database image at primary site survived the disaster. Archive and redo logs recorded since failover must be sent to the primary site and replayed against the database image there. Management Simplicity Potentially more complex to configure; both VERITAS Volume Replicator and Standby Database Replication must be configured (Oracle8i simplifies setup). Volume Replication of Entire Database Potentially higher communication cost because synchronous replication I/O is in the application response path and therefore latency is critical. Contents of a volume (usually more than one table) Any application, operating system, or database management systemcaused corruption is reflected immediately in the secondary image. Each write request incurs SRL write delay plus message round trip time (for synchronous replication only) Typically shorter. Treated as an Oracle crash recovery. Only the online redo log need be replayed. Typically longer. Image of operational database at secondary site must be created at primary site. If operational database is used by applications during restore, a switch of primary sites is also required. Probably simpler. Worst case disaster recovery scenario is approximately equivalent to Oracle local crash recovery. Table 2: Comparison of Standby Database Hybrid and Pure Volume Replication 3 Another Hybrid Replication Alternative A variation of the hybrid alternative described above is to store both online redo log and archive redo logs on volumes replicated at the recovery site. With this solution, 3 Table 2 is based on the paper Guidelines for Using Remote Mirroring Storage Systems for Oracle Database by J. Bill Lee, published by Oracle Corporation. 26

27 archive redo logs and the online redo log are both immediately available at the secondary site, so there is no possibility of unsent archive logs being destroyed in a primary site disaster. Archive logs can be applied at the secondary site by creating and mounting VERITAS Volume Manager split mirrors of the volumes containing them. This solution also allows the replaying of archive logs against the standby database to be deferred. This provides the same protection against application, operating system, or database management system-caused database corruption as conventional standby database replication with the Oracle Replication Manager managing the transmission of the archive replication logs over the network. 27

28

29 IV. Summing Up Volume replication in context VERITAS Total Data Storage Management VERITAS Volume Replicator is just one component of VERITAS Total Data Storage Management. Total Data Storage Management consists of: a solid foundation including a flexible, failure tolerant volume manager and an enterprise-class file system, automated backup and media management for distributed enterprise requirements, clustering for automated continuous application availability. Why the VERITAS Solution? Five factors qualify VERITAS uniquely to provide total data storage management solutions for the enterprise: technology leadership, product integration, multi-platform support, support, training, and service, and, experience and reputation. Technology Leadership Since the company s founding in 1992, VERITAS has been an innovator in enterprise data storage management software technology, with foundation technologies such as: high-performance, failure tolerant, logical volumes, extent-based file allocation, online volume and file system expansion and defragmentation, several frozen image technologies for volumes and file systems, direct I/O for database management systems (Quick I/O for Databases). In backup, VERITAS Block Level Incremental Backup (BLIB) has enabled fast, nonintrusive backups of Oracle databases. VERITAS clustering is also making highly 29

30 available enterprise computing a reality. Volume replication is only the latest technology in which VERITAS is innovating, to provide practical disaster tolerance. As the data storage management technology leader, VERITAS is actively driving new I/O architecture standards. Working through organizations like the Storage Networking Industry Association, VERITAS is helping to create open technologies for SAN management, for file system enhancements for effective SAN use, for increased backup efficiency, for intelligent storage devices, and in a variety of other areas. Because VERITAS is co-developing these technologies, the company will be among the first to bring their benefits to users in future data storage management products. Product Integration Product integration is the second factor that makes VERITAS the right choice for total data storage management. Examples of VERITAS product integration include: File system and volume manager cooperation that enables online expansion and defragmentation, snapshots, and Quick I/O for Databases. In addition to the database replication capabilities described in this paper, VERITAS Volume Replicator cooperates with the VERITAS Cluster Server and Global Cluster Manager to make cross-cluster disaster recoverability and global applications a practical reality. VERITAS Cluster Server agents make VERITAS File System, VERITAS Volume Manager, VERITAS Volume Replicator, and VERITAS NetBackup integral parts of a failure-tolerant data storage management environment. VERITAS develops new products and enhances existing ones with an awareness that integrated data storage management is much more compelling to users than a collection of unrelated or loosely related products. This thinking causes product decisions to be made in the context of integrated solutions. Multi-Platform Support The third reason to choose VERITAS total data storage management is the company s support of major UNIX and Windows server operating systems. With a VERITAS Total Data Storage Management strategy, the most important IT asset, skills and knowledge, is applicable across the entire enterprise. Adding or changing computing platforms does not require new data storage management skills. Support, Training and Service The fourth reason to choose VERITAS is the company s extensive network of customer support, service, and training facilities. VERITAS products are applied to some of today s most difficult IT problems, where the answers aren t always obvious. 30