Journaling and High Availability Understanding the Foundation of Logical Data Replication in IBM i Environments
Introduction The reputation and legacy of IBM's midgrange computers, from the earliest System 38 to today's Power Systems servers, is reliability. That reliability is in great part due to the journaling feature that has been present in all variations of its operating system. By journaling, that is, by keeping a running record of every transaction, every system event and configuration change, recovery from application errors or hardware problems is made easier and faster. Journaling was initially introduced as a feature of the operating system on IBM System/38 computers and was initially intended to aid technicians in any system recovery efforts. In the event of a failure, the system manager could reload the last good tape backup and then, by applying the journal entries that had accmulated since the last backup, restore the database to the point in time immediately preceding the failure. With journaling, the only transactions lost were those in progress at the instant the failure occurred. As one would expect, the recovery process was time consuming when pursued within this consruct. Retrieving the backup tapes from their offsite location and then loading the tapes and the journal entries to recover a large and complex database took several hours at best, and sometimes several days, resulting in additional downtime. Depending upon their specific availability requirements, organizations with a low tolerance for extended downtime found this recovery method inadequate. When journaling was introduced it wasn t intended to be used as an all encompasing disaster recovery agent. Yet, over time, journaling has served as the platform upon which other high availability and disaster recovery technologies were based. Subsequent improvements to journaling itself, as well as tangental technologies brought forth by both IBM and thrid party vendors, have fortified the platform s recovery and resiliency faculties and now also encompass security and compliance capabilities as well as data integrity checking. Journaling is a powerful feature, one that IBM has continued to develop and improve over the years. Yet, depending upon your business requirements, you probably still need more protection against downtime than journaling alone can provide. This white paper will cover what you need to know about journaling, what it can do and how it supports and cooperates with high availability software. 2
Journaling Basics Frank Soltis, Chief Scientist for IBM and the father of the AS/400, offers the following definition of a journal: A journal is a chronological record of changes made to a set of data. The purpose of the journal is to provide a means to reconstruct a previous version of the set of data. When a change is made to a record in a database file that is being journaled, a copy of the record is written to the journal, along with information describing the cause of the change. Relative to high availability (HA) for Power Systems servers running the IBM i operating system (formerly i5/os and OS/400), there are two basic types of journals: database journals and system audit journals. The former is the process of tracking changes to records in a data file while the latter is the process of tracking changes to object properties; i.e. object creation/deletion, authorities, etc. The two primary components of the journaling process: Journal entry the record of data changes recorded by the journaling process Journal receiver an object that holds a predefined number of journal entries. Journaling starts with a configuration of: What files will be journaled. Whether the information saved in the journal will provide one or two images. A single image configuration captures the data change after it is made. A double image configuration captures in a journal entry an image of the data or both before and after a change. What, if any, Integrated File System (IFS) objects, data queues and data areas will be included in the process. The system manager must also decide: Whether to keep the journal information on the same server where the data changes take place (" local journaling") or to also send a copy of the journal information to another partition, whether it is on the same server or on one or more additional servers (also known as "remote journaling"). If remote journaling is used, whether! the distribution strategy is one-to-one, one to many (broadcast) or one-to-oneto-one (cascade). Whether the backup system must acknowledge receiving journal entries (synchronous versus asynchronous remote journaling). The basic functionality of journaling can be broken down as follows (see Figure 2): 1. An application processes transactions that update database records. 2. The System Licensed Internal Code (SLIC), an IBM i subsystem, intercepts the transactions, records the changes that take place and creates a record of the actual changes called a journal Figure 1: IBM i Journaling 3
entry. Depending on how an application is built, there will likely be many entries for each transaction. Other information written to the journal includes the date and time of the transaction, the user identification, the initiating program, the job identification, whether the entry was generated by a trigger, the relative record number and the library of the file being journaled. 3. The newly created journal entry is written to a storage area called a journal receiver. The journal entry is now available for use. 4. Applications can create journal entries that are marked as User Generated. HA software uses this capability to generate entries when they need to serialize actions with ongoing changes to database files, IFS files, data areas and data queues. Technicians who have not worked with journaling tend to perceive it as a complex construct, but in its basic form journaling can be configured relatively easily. First you create one or several journal receivers using the Create Journal Receiver (CRTJRNRCV) command. Journal entries are first recorded in journal receivers. Next, create a journal and specify the associated receiver using the Create Journal (CRTJRN) command. Finally, you need to start journaling specific files by associating a file to a journal. This is accomplished with the command Start Journal Physical File (STRJRNPF). Local Journaling Local journaling captures changes on a primary server and writes them to a journal receiver that also resides on the primary server. Some HA software can manage the journal receiver and replicate journal records in real-time to other local databases or to logical partitions on the production server, or to a remote server for purposes of audits, security, data reconstruction, disaster recovery and HA. One advantage of local journaling (as opposed to remote journaling, described below) in the context of HA, is that it facilitates source-side filtering. Journaling records every database change, including file opens and closes (optionally), and stores other information that may not be required to maintain system availability. Using source-side filtering, HA software can avoid transmitting this unnecessary information to a backup server, thereby substantially reducing communication bandwidth, CPU cycles and disk usage. This selectivity may also reduce the total latency associated with writing the journal entries to the backup server. HA software must audit its replication processes to ensure that the primary and backup systems remain synchronized. Another advantage of local journaling is that the HA software can perform some of this data auditing on the primary machine. Thus it can detect and correct errors before sending the data to the backup machine, thereby avoiding the need to resynchronize the data after the fact. Remote Journaling Remote journaling extends local journaling by writing the journal entries to a main memory cache on the primary system and then to a journal receiver on a remote server through an IBM i-integrated transport mechanism. This copies journal entries off the primary machine as fast as possible, which is beneficial in an HA environment. The same journal entry is 4
also written to the journal receiver on the primary server. Remote journaling does not inherently include sourceside filtering. Rather, it grabs everything and, because transmission happens at the sub-system (SLIC) level, it very quickly puts data on the communication wire, thereby providing performance advantages in environments where bandwidth is not an issue. Moreover, for changes to existing records, IBM now offers an option to journal minimal data that can reduce the amount of entry-specific data sent to the journals.!figure 2: Remote Journaling Keep in mind that in addition to the process of journaling information about database and system changes, in order to achieve HA, those changes must also be applied to the backup server, in real time. Within the IBM i OS, IBM provides a journaling feature, the Apply Journal Change (APYJRNCHG) function, which can be used to apply journal entries that have been transmitted to the backup server. However, this is not a real-time process because its use requires that the databases be off-line. For the same reason, it is also limited because it will not handle many third-party software extensions that utilize user entries to send file repair, data integrity checks, object resynchronizations and so forth to the backup. Since Remote Journaling moves objects and data to the target very quickly, latency statistics show durations that are negligible. If an organization processes large amounts of data and sets it s target Recovery Point Objective at 0, then the deployment of a HA solution based on Remote Journaling is preferable. To maintain a true, resilient HA environment, HA software is required in order to apply changes to the backup database as they happen. Without such software, the journal entries will reside, unapplied, in the remote journal receiver. Finally, it is very important to note that changes for some objects, such as user spaces, are not reflected in a journal and must be handled independently to properly synchronize production and backup servers. Failure to do so may jeopardize the success of a manual or automated role swap of the production and backup servers. HA software products typically provide object-level replication services that handle these vital non-journaled objects. As stated earlier, remote journaling requires that local journaling also be active. But is there a case to be made for using only local journaling? To answer that, we will need to dig a little 5
One Size Does Not Fit All Some applications have journal dependencies that prevent remote journaling. And some businesses achieve better bandwidth and performance by being able to filter out unneeded journal entries on the source system. When evaluating HA solutions, consider these factors and, if they are issues in your environment, choose a product that provides these options and supports your applications to the level you need. deeper into how remote and local journaling work and the technical advantages and disadvantages of each. 1. When using remote journaling, little overhead is incurred on the production system by the HA processes because most of the work of harvesting (reading and saving) data from journal entries is moved to the backup environment. The work is done there using the entries in the remote journals. IBM tests have shown that executing the journal harvesting process on the primary system can result in a performance hit of up to 10% on the production environment. 1 So remote journaling is likely desirable since conserving production system resources is often a priority. 2. Even though remote journaling moves the harvesting workload to the backup system, it doesn t mean the same amount of overhead then occurs on the backup. First, there is no separate send process on the production system as this is handled by remote journaling on the production system. Second, there is no receive process above the machine interface on the backup as this is also handled by remote journaling on the backup. This adds to the resource conservation effect of using remote journaling. 3. Since IBM s remote journaling feature sends and receives data entirely beneath the machine interface, the time needed to place data on the communication wire is often substantially reduced. In tests by IBM, it took as little as 5 milliseconds to place a journal entry on the wire, even at extremely high transaction levels. 2 The reason for this speed is that the process occurs entirely within the operating system; in other words, the process has a very high priority for system processor time because it is part of the OS. With remote journaling, it is extremely rare to have any backlog of journal entries waiting to be sent to the production machine. In contrast, the local journaling functionality of some HA software may not be able to perform quite as quickly, especially if it needs to contend for CPU and I/O resources with other applications during peak production demand. This is because data-change information needs to be harvested from journal entries on the production system by the HA software, then queued for sending to the backup, and then transmitted to the backup by a separate send job. All of this happens above the machine interface and therefore has a lower priority for processor time. In addition, other processor demands on the production system can force the HA software to contend for resources, which may slow the process even further and may cause backlogs, especially when there are peaks in transaction volumes and/or high priority workloads. 4. Remote journaling has built-in auditing functions that keep track of the journal entries sent to the backup machine. If acknowledgement is not received that a sent journal entry was delivered, the process automatically resends the journal entry. This process guarantees that journaled information reaches the backup. It is important to understand how any HA software under consideration handles tracking and re-send issues. On the other hand, the functionality and features included in some HA software products can offer benefits that make them the better option in some circumstances. These benefits include the following: 1 IBM Redbook: Striving for Optimal Journal Performance on DB2 Universal Database for X-X-X-X-X-X, Chapter 6.5.6. The entire Redbook can be found at www.ibm.com/redbooks. Search under book ID#: SG24-6286-00 2 ibid, Chapter 6.2.1 6
1. HA software may prove to be very efficient because of its native code efficiency and how effectively it uses multithreading, caching and other performance-enhancing techniques. Utilizing such features and functionality may make the HA software s local journaling methods the preferred choice. 2. Relying on IBM s remote journaling in the absence of a comprehensive HA solution construct may mean foregoing advanced object replication features found in some HA 1 software products. There are more than 60 objects types that an HA solution should be able to replicate to the backup system. A high availability solution must have the capacity to handle objects that server both application and OS functions like security objects and user profiles. 3. Remote journaling may guarantee that a journal entry makes it from the local machine to the remote journal, but, because it is not involved in applying entries in that journal to the backup files and databases, remote journaling, on its own, cannot provide or guarantee backup data integrity. HA software is required to apply the transactions in real time and to audit the integrity of the replication and synchronization processes. When the HA software uses a proprietary transportation mechanism, it may also be able to run some of the integrity-checking processes on the production machine, verifying and, if necessary, correcting errors in the data and objects before they are shipped to the backup system, rather than shipping over bad data and then verifying and resending it after the fact. Synchronous versus Asynchronous Remote Journaling Remote journaling transmits journal entries either asynchronously or synchronously. For reasons of speed, performance and system overhead, asynchronous journaling is more common as it allows local processes to continue without waiting for confirmation that journal entries have been received by and applied to the remote journal. On the other hand, synchronous remote journaling is the better approach if absolutely every journal entry positively must reach the target, leaving no backlog of in-flight transactions. With synchronous remote journaling, the journal entry is transmitted to the remote server's main memory cache and an acknowledgement is sent back to the primary server. At that point, and only at that point, the original transaction is allowed to continue, thus facilitating a write to the primary journal receiver and, subsequently, to the primary database. Independently, the backup server also writes the entry from its memory cache to its journal receiver. The reason why synchronous remote journaling is not used universally, despite being more reliable, is that the process of getting an acknowledgement that the journal entry has arrived safely in main memory cache on the target server consumes system cycles and bandwidth, defers transaction commitment, and may keep users waiting. Synchronous remote journaling does not, by itself, guarantee synchronization of the primary and backup servers. The entries residing in the primary and backup journal receivers are confirmed, but not the tables in the primary and backup databases. Since the journal entry is written to a main memory cache on the target server first, transaction updates to the source database are not committed to disk until the journal-entry write to main memory cache on the backup has been acknowledged. 7
! Figure 3: Synchronous Remote Journaling It is also important to point out that synchronous journals are processed serially. The journal entry write acknowledgement from the backup comes from journal-entry writes made to the remote main memory cache (see Figure 3). This improves overall processing performance, but it also means that the remote journal receiver does not reflect the primary journal receiver (which is updated following the acknowledgement) until some time in the future. Thus, depending on the backup server workload and the timing of a server failure, the backup server may not be a true replica of the production server. Currently, HA software is required to ensure backup validity under these conditions. Since primary server journal entries are written to the primary journal receiver, and subsequently to the primary database, only after receiving an acknowledgement from the backup server, any transactions that are yet to be confirmed or acknowledged when a primary server fails will not be in the primary journal or database. HA software must deal with these synchronization issues by reading the unconfirmed entries from the target journal. Complications are not limited to primary-server failure. If the backup server should abruptly fail, journal entries residing in its main memory cache will have not been written to its journal receiver or, therefore, to the backup database. In such a case, when remote journaling is restarted after the backup server comes back online, the remote-journal entries residing in main memory cache will be synchronized with the primary journal and the HA software can continue applying the journal entries to the backup database. 8
Improving Remote Journaling HA Performance Remote journaling is often viewed as all or nothing, with all journaled files being sent to the backup server. There is no provision for source-side filtering or other journal management and this impacts bandwidth. IBM provides a journal minimal data option that journals only the changed portions of data rather than complete records. This can reduce bandwidth requirements, provided that the applications typically update only small segments of data records. In contrast, when transactions change substantial portions of each table element or add records, this feature offers little benefit and, because of the overhead associated with journal minimal data, may even add to bandwidth and processing requirements. The journal minimal data option raises another concern the format of the entries. Previously, programmers could read and decipher journal entries reasonably easily. This is not the case with minimal-data entries, which may create problems for programmers and for userwritten programs that process journal entries. Journal minimal data can be used with remote journaling, but it can also be used with HA replication software, which adds the advantage of source-side filtering. Even when using journal minimal data, an investment in bandwidth may be required to maintain acceptable performance levels. It is necessary to define metrics relative to transaction requirements, and then evaluate the tradeoffs of local versus remote journaling and the costs of any proposed solution. This calculation can be complex as IBM i compresses the information stored in the journal and remote journaling utilizes this compressed form to send entries, whereas HA replication software must read the data in uncompressed form before doing any filtering. If using remote journaling, it is important to specify RCVSIZOP (*RMVINTENT), as this causes the internal entries needed by IBM i journaling to be stored in a separate internal area and, if possible, using separate disk arms. These entries are not then sent to the target remote journal as their use is primarily for single system recovery. When large quantities of small transactions are being generated on the production system, it is important for the journaling and send processes to be able to keep pace. With asynchronous remote journaling, should the journaling process begin to fall behind, the number of packets being sent is reduced somewhat by bundling journal changes together until it catches up. Some HA software packages also have the ability to bundle many transactions into a single block of data for a single send. The efficiency and effectiveness of these features should be compared against your particular requirements as you plan your HA environment. There are other journaling features remote and local available in IBM i, such as standby journaling, journal caching and more, that can be used to enhance journaling performance. These features should be considered when planning you're HA solution as they may help to more fully optimize DASD and replication efficiencies. 9
Comprehensive Journaling The system audit journal captures information about system tasks, activities and changes to non-database objects. This function is critical to ensuring that backup servers are ready to take over should the primary server fail. If information about updates to non-database objects is processed via a remote journal, it will then reside on the backup server s journal. Complex logic is necessary to determine what needs to be done with this information. This typically involves communicating back to the source server to retrieve the whole object. In contrast, by processing the journal on the source server, HA software can directly replicate the object to the backup server as required. Thus, because journaling cannot capture the actual content changes to objects in the audit journal, remote audit journaling will probably prove counterproductive. When evaluating HA options, be certain that your selected software replicates all data types and objects. Using remote journaling to replicate user data (DB2 databases, large objects, IFS, data areas and data queues) is comparatively easy. However, if the backup server is to be complete and ready to run critical business applications when called upon to do so, the HA solution must also replicate application objects (programs, user profiles, authorization lists, configuration objects, spooled files, etc.). Synchronization and Integrity Issues The fundamental role of an HA solution is twofold: 1. To provide quick operations recovery in any unplanned downtime scenario and 2. To provide operations continuity during planned downtime. Accordingly, system synchronization (the assurance that all transactions that have been applied to the primary database have also been applied to the backup database) and data integrity (the ability to verify that the contents of the backup database are the same as the contents of the primary database at any given time) are fundamental. If the data and objects on a backup server are not identical to those on a primary server, this might make it impossible to swap systems when required. If so, the HA environment is worthless. IBM i journaling provides neither source and target synchronization nor integrity validation. This functionality is provided only by advanced HA software. Without it, automated failover or manual switchover (the act of substituting the backup for the primary server) cannot safely occur. Finally, remote journaling doesn't provide a mechanism for removing journal entries from the remote server once they are applied to the database. Entries prematurely designated for removal would result in a data-integrity or synchronization problem as these transactions would not be available for application. Some HA products provide journal-manager tools that coordinate with their journal-apply processes so that journal entries are deleted only after the HA software is finished with them. These products can coordinate the deletion of both remote and local journal entries. 10
Journaling s Most Recent Enhancements Conclusion In February of 2008, IBM unveiled a new release of IBM i. V6R1 incorporates a broad range of new features and capabilities, including several that have an impact on journaling. Many of these new features evolved from suggestions made by IBM Business Partners who develop HA solutions. One important addition made by IBM is the ability to journal entire libraries in V6R1. Changes made to a library can now be journaled based on user-defined rules. IBM i can automatically journal objects created in, copied to or restored into a library based on user-specified inheritance rules. End Journal While Open is another feature that has an impact on HA technology. Occasionally, when HA products are initially activated, all files are inadvertently journaled. This is not recommended since it can degrade performance. End Journal While Open eliminates the closed/active file restriction so journals can be terminated more quickly. Additionally, moving objects from one journal HA is a Solution, Not Software to another in order to balance or tune an Be sure that a fast installation time doesn t require you to figure HA workload encompasses fewer steps. out exactly what needs to be replicated and then perform all of the journaling and communications set up. Look for solutions that New remote journal status screens have automate setup processes for you. An HA vendor should also been added to simplify the administration of HA tools that are based on figuration, and then optimize the solution to protect your applica- help you figure out what needs to be replicated, complete the con- remote journaling technology. As noted tions as efficiently as possible. The best solutions replicate only earlier, Remote Journaling creates an what s needed, not everything. That saves unnecessary processor identical instance of a journal receiver and communication overhead. Automation and expert guidance on a remote system so the HA solution ensure you re fully protecting your business and lower the total can apply changes to objects on the cost of ownership. remote system by reading journal entries directly from the remote journal receiver. In V6R1 IBM added status information for remote journals that is accessible in new status screens. Prior to the addition of this feature it was difficult to assess how far behind the remote journal was in transmitting journal entries to the remote system. A new command has been added in V6R1 to facilitate the replay of specific individual journal entries using raw journal-entry data. HA solution vendors use this API to replay certain changes from a remote journal. The QjoReplayJournalEntry API replays data-area and dataqueue creation journal entries. Another enhancement that has an impact on HA facilitated by journaling is a new Cache Flush Feature. To address cashing staleness, IBM has added a control to adjust a cache s staleness value. Replication speed and latency, filtering requirements, and server capacity and network overhead are all areas that need to be considered when gravitating toward one journaling method or the other. Regardless of the type of under consideration, journaling, in and of itself, is not a replacement for a carefully orchestrated availability strategy that incorporates 11
HA technology. Several other features, functions, and services are needed to accompany journaling to ensure that redundant applications and timely data sets are available when needed. Some of these features facilitate: Automation of as much of the installation and configuration processes as possible. Real-time status monitoring of the primary and backup systems. Help with the management of journals. Multiple possible transport methods for each transaction. Application of the journal-entry writes to the backup database. Maintenance of critically important server synchronization and data integrity between the primary and backup servers and their databases. Integrated auditing and autonomics to ensure switch-readiness and synchronization accuracy. Seamless role swaps for servers in planned and unplanned scenarios. Big decisions relating to IT infrastructure typically call for good measure of due-diligence. What s true for ERP or CRM is true for HA: Do your homework to understand each component of a solution. Fully recognize your business requirements and apply the appropriate technology, methods and skills to achieve a solution that addresses your specific availability needs and provides a satisfactory ROI. Easy. Affordable. Innovative. Vision Solutions. Vision Solutions, Inc. is the world s leading provider of high availability, disaster recovery, and data management solutions for the IBM System i and System p markets. With a portfolio that spans the industry s most innovative and trusted HA brands, Vision s itera, MIMIX, and ORION solutions keep business-critical information continuously protected and available. Complementing Vision s availability offerings, Vision Director delivers a highly integrated set of applications that proactively monitors, manages and optimizes System i servers, databases and application environments to help ensure the continued health of System i servers. Affordable and easy to use, Vision products help to ensure business continuity, increase productivity, reduce operating costs, and satisfy compliance requirements. Vision also offers advanced cluster management, data management, and systems management solutions, and provides support for i5/os, Windows and AIX operating environments. As IBM s largest high availability Premier Business Partner, Vision Solutions oversees a global network of business partners and services and certified support professionals to help our customers achieve their business goals. Privately held by Thoma Cressey Bravo, Inc., Vision Solutions is headquartered in Irvine, California with offices worldwide. For more information call 801-799-0300 or toll free at 800-957-4511, or visit. itera ORION MIMIX SOLUTIONS 17911 Von Karman Avenue, Suite 500 Irvine, CA 92614 800-957-4511 801-799-0300 Copyright 2008, Vision Solutions. IBM and System i are trademarks of International Business Machines Corporation. WP_Journaling_E_0809