High Availability Essentials Introduction Ascent Capture s High Availability Support feature consists of a number of independent components that, when deployed in a highly available computer system, result in robust, fault-tolerant operation. Although fault-tolerant systems are primarily intended for very large enterprise scale organizations, even the smallest company may benefit from some of these components. Note Many high availability features apply to both Ascent Capture and Ascent Capture Internet Server. For convenience, in this appendix only, the term Ascent Capture will be used to include both Ascent Capture and ACI Server, unless otherwise noted. This appendix includes basic information about fault-tolerant systems in general, specific information about Ascent Capture s support for high availability, and specific installation guidelines. Tip Configuring and optimizing a fault-tolerant system can be a complex, timeconsuming and expensive task. This appendix includes only information related to Ascent Capture, and it is not intended to be a complete analysis or guide to creating such systems. Depending on the size and complexity of your installation, as well as the skills and experience of your staff, it may be best to retain a consultant who specializes in high availability systems. What is High Availability? A high availability (resilient) system, at the most fundamental level, is capable of performing its tasks with virtually no downtime. Any complex system has many components, any of which can fail or degrade at any moment. A highly available system will continue to function even if one or more of its components fail. Installation Guide for Ascent Capture and Ascent Capture Internet Server 155
The key idea behind high availability is that a system is only as good as its weakest component (also known as a fault domain). A system is made more resilient by providing redundancy for those fault domains most likely to fail, starting with the weakest and moving up. Ascent Capture and High Availability Ascent Capture is only one link in the chain of high availability. It has features that are intended to work within, and take advantage of, a high availability computer system. Some of these features, even if used in a normal environment, will nonetheless improve the performance and reliability of Ascent Capture. Common Terms You may find it helpful to familiarize yourself with the following terms, which are used throughout this appendix. Failover In case of a problem, failover is the act of automatically switching to redundant or standby equipment upon the failure or abnormal termination of the currently active equipment. Failover happens without human intervention. Scalability Scalability is the ability to respond to increased demand by adding additional equipment, processes, or other capabilities. Recover (a batch) In case of a problem with a batch, recovery is the process of restoring the state of the batch to whatever it was prior to the batch being opened, and recovering all changes made to the batch that would otherwise have been lost due to the problem. After recovery, the batch becomes available to Ascent Capture applications running on any station, including the crashed station. In many cases, recovery happens invisibly to the user (refer to Automatic Batch Recovery on page 160 for details). Rollback (a batch) In case of a problem with a batch, rollback involves only restoring the state of the batch to whatever it was prior to the batch being opened. Changes made to the batch may be rolled back to what the data was prior to the batch being opened. Rollback occurs if the Batches in SQL Server database option is not enabled and the batch is recovered by a process running on another machine. After rollback, the batch becomes available to Ascent Capture applications running on any station, including the crashed station. In many cases, rollback happens invisibly to the user (refer to Automatic Batch Recovery on page 160 for details). 156 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials What Makes Ascent Capture Resilient? Ascent Capture incorporates a number of individual components that allow you to select a degree of fault-tolerance that is suitable for your business. Because Ascent Capture s degree of resilience can be readily scaled, organizations of any size or complexity can find a level of high availability that matches their needs and budget. To ensure that an Ascent Capture installation meets the criteria for high availability it should: Use Microsoft Cluster Services. Run multiple license servers, providing automatic failover. Use Web farms for installations with ACI Server (based on Microsoft Network Load Balancing). Built-in automatic batch recovery of in-progress batches if a workstation or application fails. Run multiple instances of all the Ascent Capture modules. If one instance fails the other instances can continue working. Also known as scalability. Combined, these steps allow Ascent Capture to continue processing batches even if specific points in the system fail. Figure E-1 shows an example of an Ascent Capture high availability configuration, including ACI Server remote sites. Installation Guide for Ascent Capture and Ascent Capture Internet Server 157
Figure E-1. Ascent Capture High Availability Configuration Example 158 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials Microsoft Cluster Services Microsoft Cluster Services (MCS) provides the ability for a group of independent servers to be managed as a single system for higher availability and greater scalability. A minimum configuration would consist of two servers connected by a network, a mechanism for sharing their disk data, and cluster software (in this case MCS). This software provides failure detection and recovery, as well as the ability to manage the servers as a single system. MCS also provides software routing of incoming network requests to a shared IP address. With Ascent Capture, performance and throughput do not scale as more nodes are added to the cluster, since each cluster resource can be active on only one node at a time. Note MCS does not protect a system from application failures on a particular machine (where the application crashes, but the machine continues to run). However, SQL Server Enterprise is cluster aware and automatically forces a failover if one of the SQL Server applications fails (even if the machine continues running). Although Microsoft Cluster Services and SQL Server Enterprise provide automatic failover, client applications may receive errors during the time needed to complete the failover process. While Ascent Capture hides most errors, some may be exposed to the user. Note Ascent Capture supports MCS only on Microsoft Advanced Server 2000 or Microsoft Windows Server 2003 Enterprise Edition. License Server Failover Ascent Capture uses a license server to manage all licensing requests. The license server uses a hardware key to lock the activation code to a particular serial number and to store volume usage. Ascent Capture supports multiple license servers a primary and one or more backups. When multiple license servers are available, each license server needs its own hardware key, activation code, and license configuration. Note that volume usage in the backup license servers can be very low (there only needs to be enough volume to last until the primary license server is restored). Installation Guide for Ascent Capture and Ascent Capture Internet Server 159
If the primary license server fails, the system automatically switches to another license server. The system can use this backup license server until it runs out of volume. When the primary system is restored, Ascent Capture automatically resumes using it. There are no performance gains from installing multiple backup license servers, so there is no performance-based reason to have more than one backup license server. To the contrary, the introduction of multiple backup license servers requires proportionally more work by the system administrator. If you plan to have more than one backup license server, you might want to weigh any perceived benefits against this additional overhead. Automatic Batch Recovery Ascent Capture can automatically recover a batch if a failure occurs while the batch is being processed. The Automatic Batch Recovery feature is built into Ascent Capture and does not require any special installation or configuration. In order for Ascent Capture to detect a failure and recover batches, some other Ascent Capture application must be running. The other Ascent Capture application will eventually detect the failure and initiate the batch recovery process. This discovery and subsequent recovery will usually take no longer than 15 minutes, and in many cases significantly less time (a matter of seconds). The delay depends on the exact timing and sequence of events. If an Ascent Capture application detects a failure on another station and restores a batch to its last known good state, changes made to the batch after it was opened on the crashed station may be lost if you do not have your batches in SQL Server. The ability of Ascent Capture to completely roll back or recover a batch without data loss depends, in part, on the batch database option you have selected. If the Store batches in SQL Server option is enabled (online help keywords: Database Utility > Database tab), a failed batch is set (rolled back) to its last good state, and then becomes available to Ascent Capture applications. In the vast majority of cases, no data is lost. If the Store batches in SQL Server option is not enabled, no data will be lost if the batch is opened on the same workstation where the problem occurred. If the Store batches in SQL Server option is not enabled, data will be lost if the batch is opened on some workstation other than the one where the problem occurred. 160 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials Batch recovery and rollback events are logged to the Ascent Capture error log (Err_YYMM.txt). Note If you discover a problem before a batch is automatically recovered, you may decide to attempt to initiate recovery of the batch by manually restarting the application. If at all possible, you should restart that application on the same station. This maximizes chances for a full recovery of the batch. ACI Server Web Farm Support ACI Server supports the Microsoft NLB (Network Load Balancing) Web farm technology with multiple nodes (servers). Network load balancing allows a Web server-based application (such as ACI Web Server components) to distribute its workload across multiple Web servers. This not only improves performance, but also provides failover support because if one Web server fails, other Web servers in the network will take over. Such configurations are known as Web farms. Installation Guide for Ascent Capture and Ascent Capture Internet Server 161
Configuring Ascent Capture for High Availability This section includes instructions for configuring Ascent Capture for high availability. You can skip those steps that do not apply to your installation. Note that the Automatic Batch Recovery feature is an integral part of Ascent Capture, so there are no specific steps required to implement or configure it. Step 1 Step 2 Step 3 Install and configure Ascent Capture. It is recommended that you initiate the installation of Ascent Capture Server from one of your cluster nodes. Set up one or more backup license servers. Perform this step only if you intend to install one or more backup license servers. In general, there is no advantage to having more than one backup server. Set up your Web farm for use with ACI Server. Perform this step only if you want to use Microsoft NLB to set up a Web farm for use with ACI Server. Step 1: Install and Configure Ascent Capture in a Microsoft Cluster Server Environment Before you install Ascent Capture in a high availability environment, you must satisfy the following prerequisites: Configure Microsoft Cluster Services. For specific installation details, refer to your Microsoft documentation. Note The computers hosting the Ascent Capture server files, as well as their network connections, must be as optimized as possible. For high availability installations using Microsoft Cluster Server with a redundant drive file system, certain drive configurations may slow down performance while increasing resilience. Install SQL Server on the cluster. For specific installation details, refer to your Microsoft documentation. Create the Ascent Capture shared folder from within the Cluster Administrator. This folder will be used for the Ascent Capture Server Files and so it requires Full Control permissions. Refer to Server files on page 21 for more information about the shared folder. Do not share the folder in Windows Explorer as you normally would for a file share. 162 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials Note High availability goals may conflict with the desire for optimal performance. For example, RAID drives often provide greater availability, but perform more slowly than standard drives. Additionally, you may prefer to install multiple applications on your ACI Service computers to reduce cost. You need to weigh all of these issues when configuring or optimizing your system. The following steps provide a recommended installation procedure for Ascent Capture that will take full advantage of a high availability environment. 1 Install Ascent Capture as you normally would. 2 When asked to specify a server files folder to store Ascent Capture data files, be sure to use the file share you created as one of the prerequisites above. 3 After installing Ascent Capture, run DBUtil.exe and change the Ascent Capture database configuration so that it uses the SQL Server instance installed on the cluster. Then, if desired, enable the Store batches in SQL Server option. 4 All external, shared files must be placed in the cluster file share folder. These files include: Image Files The folder holding image files is configured with the Image folder setting from the General tab of the Batch Class Properties dialog box in the Ascent Capture Administration module. OCR Full-Text Dictionary This dictionary is configured with the Dictionary setting on the OCR tab of the Document Class Properties dialog box. PDF Dictionary This dictionary is configured with the Dictionary setting on the PDF tab of the Document Class Properties dialog box. Field Type Dictionary This dictionary is configured with the Dictionary setting on the Field Type Properties dialog box. 5 By default, when file operations fail, they are retried every 0.5 seconds for a total retry time of 2 minutes. You can configure these parameters by updating the ACConfig.xml file in the Ascent Capture Server files folder, Config subfolder. The ReliableFileSystem element must be added as a direct child of the ACConfig element. <ACConfig> <ReliableFileSystem> <Retry MaxRetrySeconds= 120 MilliSecondsBetweenTries= 500 /> </ReliableFileSystem> </ACConfig> Installation Guide for Ascent Capture and Ascent Capture Internet Server 163
Step 2: Set Up One or More Backup License Servers Ascent Capture includes a single license server, which is automatically installed. To make the system more resilient, you can start multiple license servers on additional client workstations, provided you have obtained the required licensing. You may optionally run other server-based applications (such as the OCR Full Text or Release modules, but not the Scan or Validation modules) on the same computer as the license server. However, overall system performance may be reduced. This is because the license server requires frequent network transmissions, low to medium CPU usage, and occasional disk access. It is usually easier to configure a successful system by keeping the license servers on dedicated computers. However, running server applications together with the license server can be accomplished through careful configuration. Note If you install your license server on one of the cluster node computers, it is strongly suggested that you have a backup license server. Otherwise, you will lose your failover capability if that node fails. To configure a backup license server 1 Install an Ascent Capture Workstation on a second (or additional) machine. 2 Install the backup license server s hardware key. 3 Run ACLicSrvCfg.exe from <AC Workstation Installation Folder>\bin\ ACLicSrvCfg.exe. 4 Leave the default user set to Local System and click OK. Note If you are also using ACI Server on this machine, a different user is necessary. This user must have access to the folder specified for the licensing service, as well as to the needed ACI Service resources. 5 The utility starts the service, and then checks to see if the service is running properly. 6 Ensure that the license on the primary server has already been activated (this should have happened when you installed Ascent Capture). 164 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials 7 To activate the backup server, invoke the license utility with appropriate command-line parameters that cause it to connect to the backup license server: <AC Workstation Installation Folder>\bin\ AcLicUtil.exe /S <ServerName>. For example, to run the License Utility against the license server named LicBackup you might use this command prompt: c:\program files\ascent\bin\aclicutl.exe /S LicBackup. 8 Finally, use the License Utility to activate your backup license server. Step 3: Set up your ACI Server Web Farm If you want to use an ACI Server central site with a Web farm, use the following procedure. Following the procedure, each step is explained in more detail. The Ascent Capture system must include at least two Ascent Capture workstations that host ACI Services. The system must also include at least two ACI File Cache folders on any two machines. Before you install your ACI Server Web farm, you must satisfy the following prerequisites: You must have two or more computers without Ascent Capture installed on them. These will function as the Web farm. You must install at least two Web servers on at least two of these computers. You also need to install IIS. Refer to your IIS documentation for details. Note WebSphere is not supported for use with Ascent Capture in a Web farm. You must configure these Web servers as part of a Web farm using Microsoft Network Load Balancing (NLB). The specific Web farm technology used by ACI Server is Microsoft NLB. Refer to your Microsoft documentation for details. When setting up the NLB port, the port range must include the port for which the Web server is configured. By default, the ACI Web Server listens on port 80. Installation Guide for Ascent Capture and Ascent Capture Internet Server 165
Note Only IIS and ACI Web Server components are allowed on the Web farm nodes. You cannot install any Ascent Capture workstation components, ACI Server file caches, or ACI Services on them. To set up an ACI Server Web farm 1 Enable the first ACI Service. 2 Configure the Web servers as part of a Web farm. 3 Install ACI Web Server components on the Web servers. 4 Enable additional ACI Services. 5 Configure ACI Server Cache Folders. Note The Ascent Capture server, the ACI Server cache folder(s), and all Web servers in the configuration must belong to the same domain. Enable the First ACI Service Next, you must enable the first ACI Service at the central site. Refer to Chapter 8 Installing ACI Server on page 71 for information about installing ACI Server (and enabling the ACI Service). Note In subsequent steps, you will change the default ACI Server file cache folder and create new ones on each ACI Service machine. If installing ACI Server for the first time, you can skip the parts about configuring the cache folder. Refer to Chapter 8 Installing ACI Web Server Components on IIS on page 76 for information about the cache folder. Install ACI Web Server Components on the Web Servers You need to install the ACI Web Server components on each and every Web server in the Web farm. Note The same Ascent Capture Server and the same cache folder location(s) must be used for all Web servers. Refer to Chapter 8 Installing ACI Web Server Components on IIS on page 76 for information about installing these components. 166 Installation Guide for Ascent Capture and Ascent Capture Internet Server
High Availability Essentials Enable Additional ACI Services You need to enable at least one additional ACI Service on an Ascent Capture client workstation. If desired, you can add more ACI Services. This not only provides greater reliability, but may also improve performance. Refer to Chapter 8 Step 2: Enable the ACI Service at the Central Site on page 74 for information about installing ACI Services. Note You cannot install ACI Services on the Web servers. Configure ACI Server Cache Folders By default, all ACI Services use an ACI Server cache folder located within the Ascent Capture Server files folder. Assuming that you have implemented Microsoft Cluster Services as described in this appendix, this folder will be highly available. However, because there are additional ACI Services deployed on workstations, then the default ACI Server File cache folder should be disabled and a local cache folder added on each client workstation running an ACI Service. Each local cache folder must be accessible by all ACI Services. This will reduce network overhead and may improve performance. Refer to Chapter 8 Configuring Additional File Cache Folder Locations on page 85 for information about adding multiple cache folders. Remember that it is necessary to run the ACI Server Wizard on all the Web servers after making the changes. Note If desired, you can disable the original cache folder. To do so, edit the ACConfig.xml file by replacing the path to the original cache folder with the path to one of the added cache folders. Optimize Your Web Farm With NLB, client requests are load-balanced among the different Web servers. Adding additional Web servers theoretically improves performance, if the current Web server is the bottleneck. In order to obtain these performance benefits, all the components involved with ACI Server batch upload and insertion must be optimized. In addition, you should ensure that all points in your network have sufficient bandwidth to handle your projected loads. Installation Guide for Ascent Capture and Ascent Capture Internet Server 167
High Availability for Smaller Installations using Microsoft Cluster Server If you are able or willing to accept a somewhat diminished level of high availability and performance, you can use some of the following configuration variations to reduce the number of machines you need. These variations represent a trade-off between high availability, performance, and hardware costs. You can install Ascent Capture on one of the cluster machines. If you decide to do this, you should use the node that has the least load. Install the Ascent Capture workstation on the remaining cluster machine using the workstation installation program (setup.exe in the WrkInst folder). Note During the installation of Ascent Capture, be sure that the other cluster machine is powered on. This is required to ensure that the shared disk resource is available during the installation. Keeping in mind that this will have a negative impact on performance, you may choose to run any of the following Ascent Capture server applications on the cluster computers. ACI Service Custom modules with no user interaction (such as Xtrata) License server OCR Full Text module PDF Generator module Recognition module Release module Remote Synchronization Agent XML Auto-Import 168 Installation Guide for Ascent Capture and Ascent Capture Internet Server