DataXchange Enterprise Performance Whitepaper

DataXchange Enterprise Performance Whitepaper

Disclaimer All rights to trademarks or service marks mentioned within this document belong to their respective manufacturers or organizations. E-Z Data has made every effort to make this report as complete and accurate as possible, but no warranty or fitness is implied. The information is provided on an as is basis. E-Z Data shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained within this report. Copyright 1998-2006, E-Z Data, Inc. All Rights Reserved No part of this documentation may be copied, reproduced, or translated in any form without the prior written consent of E-Z Data, Inc. Contact E-Z Data, Inc. 918 East Green Street Pasadena, CA 91106 Telephone: (626) 585-3505 Fax: (626) 440-9097 U.S. toll-free fax: (800) 779-3123 http://www.ezdata.com

Table of Contents 1. Executive Summary...1 2. Data Movement - Challenges...2 3. Scalability Exercise...2 3.1. Overview...2 3.2. Technology...2 3.3. Configuration...3 3.4. Data...3 3.5. Testing Methodology...4 4. Test Results...6 5. Analysis of Results...8 5.1. DataXchange Enterprise: Highly Scalable...8 5.2. Fine-tuning...8 5.3. The Impact of Differing Requirements...8 5.4. Lessons Learned...8 6. endix...9 6.1. DataXchange Enterprise (DXE) Server...9 6.2. Components of DataXchange Enterprise Server...9

1. Executive Summary DataXchange Enterprise Server is a tool designed to efficiently move legacy and third-party data into SmartOffice, E-Z Data s integrated management platform for financial professionals. To fully scrutinize the robustness of DataXchange Enterprise Server, we recently conducted a five-week joint scalability exercise at the IBM scalability labs in San Mateo, California. The study was specifically designed to test DataXchange Enterprise s performance and scalability, and to fine-tune both the application and the database to ensure optimal client results. The tests confirmed DataXchange Enterprise s excellent scalability and proficiencies supporting data volumes from even the largest financial services organizations. We worked closely with IBM s DBAs, as well as WebSphere and AIX tuning specialists, to evaluate and tune DataXchange Enterprise in order to identify ideal configurations for companies requiring high volume data processing. The results clearly demonstrate that DataXchange Enterprise can predictably handle the millions of records needed to support even the largest organizations. In the lab environment, DataXchange Enterprise was used to process 16.8 million policy records, 3.5 million investment records, and 18 million contact records, and was shown to be capable of processing 1 million policy records per hour, and 1.1 million investment records per hour. These impressive test results confirm DataXchange Enterprise s ability to support large-scale deployments by moving financial services data quickly, efficiently, and without error. Additionally, the information from this IBM testing can help simplify the process of budgeting for required hardware and planning accurate data processing periods when planning deployments of E-Z Data s SmartOffice and DataXchange Enterprise solutions. Page 1

2. Data Movement - Challenges SmartOffice is E-Z Data s comprehensive Web-based Contact Relationship Management platform for financial professionals, delivering cross-enterprise solutions for data management and sharing. One of the significant features of this application is its ability to create value from the combination of legacy and field-entered data. However, to achieve this value, there must be a method to efficiently exchange large amounts of data to and from the SmartOffice application. Large financial services organizations face many common challenges regarding data movement. Some of the most common issues include: Needing to process extremely large amounts of data in a very short window of time, as data from third-party clearinghouses is usually not available to the carrier until late evening or early morning. Having to be able to transfer and store complete data sets, not just deltas, from each extract, exponentially increasing data volume. Extracting data for the entire office, though a greater strain on systems, is easier than identifying which advisors use which software during extracts. Easily expanding to meet the needs of additional business groups wanting to have their data contained within the application. Integrating with myriad legacy systems that contain existing policy data. 3. Scalability Exercise 3.1. Overview The most recent DataXchange Enterprise performance and scalability test was conducted from July 31 to September 4, 2006, at the IBM Innovation Center in San Mateo, California. The results of the test were to be used to fine-tune the application and the database to ensure optimal results. A review of the various aspects of the testing process, including the technology, configuration data, and methodology displays below. 3.2. Technology E-Z Data and IBM worked in close partnership to test the scalability of the DataXchange Enterprise server on IBM s AIX, UDB, and WebSphere platforms. Hardware / Software Server Hardware Software Web IBM pseries 570, 16GB RAM, 4x1.9Ghz AIX 5.3, IHS (IBM s http server) IBM pseries 570, 16GB RAM, 4x1.9Ghz AIX 5.3, WebSphere lication Server DB IBM pseries 570, 32GB RAM, 16x1.9 Ghz AIX 5.3 UDB 8.3 Page 2

3.3. Configuration 3.4. Data Below is the amount and type of information used during the scalability test. Table Rows Description INTERESTPARTY 80,316,966 Interested Parties linked agents, owners, insureds, etc. CONTACTACCTMASTER 19,557,898 Interested parties for accounts. CONTACT 18,109,825 Individual (person) contact records ASSIGNMENT 18,109,814 Record assignment table used for security PERSONAL 18,109,786 Personal details about contact records PHONE 17,649,757 Phone numbers ADDRESS 17,534,906 Addresses New business history records, policy status history information NBHISTORY 17,518,912 POLICY 16,876,220 Top level object record for all types of policies LIFE 11,072,566 Detail information regarding life insurance policies RIDER 10,554,182 Information on policy riders USERROLE 3,748,304 Predefined user role for an application user record POSITION 3,574,699 Investment position information FA 3,574,698 Fixed Annuity information ACCTMASTER 3,438,328 Security Account Master information BUSINESS 2,532,678 Individual Business (not a person) contact records LICENSENUM 2,502,899 Agent license information AUTO 1,839,351 Auto policy information Fig 2: Tables with more than 1 million rows Page 3

Data Distribution Many factors affect the distribution of the data including how many policies are included per contact record, how many riders there are for each policy, how many addresses and phone numbers exist per contact, etc. In order to complete the test using the most realistic data that includes an accurate distribution of records, E-Z Data worked with a carrier to use live data that had been scrambled to prevent any personally identifiable information from remaining visible during testing. In this way, the identities of the actual records were protected while the data distribution pattern was maintained. 3.5. Testing Methodology Testing at the IBM scalability labs is structured to enable a controlled test environment in order to deliver the most accurate results. The DataXchange Enterprise tests were structured to simulate actual usage in financial services data-movement business scenarios. IBM and E-Z Data employed the following testing methods to simulate a real world test. Terminology Throughput of the DataXchange Enterprise process is measured in terms of TLOs (top-level objects) per hour. Policies or Investments are examples of TLOs. Each TLO consists of multiple sub-objects. For instance, a policy has the following sub-objects: 1. Interested Party Interested parties linked agents, owners, insureds, etc. 2. Contact Individual (person) contact records 3. Life Detail information regarding life insurance policies 4. Rider Information on policy riders Testing Variables To study the impact of a variable on performance, we minimized the changes to the number of variables between test cases. For instance, between test cases 1 and 2, the database server central processing units (CPUs) were increased from two to four, and the commit threads were increased from 12 to 24. Similarly, between test cases 2 and 3, the application server CPUs were increased from two to four, and the commit thread increased from 24 to 28. Ideally, we would have liked to have changed only one variable between the test cases to understand their true impact. However, because of the number of permutations and combinations available, we elected to change the minimum practical number of variables (not just one) between the tests. The main factors affecting the throughput of DataXchange Enterprise include: 1. Database servers CPUs 2. lication servers CPUs 3. DataXchange Enterprise tuning parameters a. Pre-process thread counts b. Commit thread counts Additionally, it is known that the storage system used by the database also affects the performance and throughput of DataXchange Enterprise; however, in this test, the storage subsystem was not considered a variable as all the tests were run on the same storage system. Page 4

Important Testing Note The numbers seen in the resulting data are based on updates. Once the data integration is implemented, DataXchange Enterprise will most often be updating policies and investments, a much less systemintensive task. Test Case Load Type CPU Combination (DB-) Commit Threads (PP - Commit) 1 Policy 2 DB, 2 12 PP, 12 C 2 Policy 4 DB, 2 24 PP, 24 C 3 Policy 4 DB, 4 24 PP, 28 C Page 5

4. Test Results CPUs Threads Load Size DB APP PP Commit TLO per hour Policy Data Processing RPH Time (in Minutes) DB during PP DB during Commit CPU Usage during PP during Commit ~500K 4 2 12 12 338,000 338,000 90 87 95% 87 95% 60 75% 60 75% ~500K 4 2 24 24 599,386 599,386 49 40 92% 92 100% 1-10% 20 95% ~500K 4 4 24 28 608,000 3,769,640 50 65 90% 65 90% 65 82% 65 82% ~507K 6 4 36 90 760,735 760,735 40 40 80% 85 98% 1-3% 75 98% ~500K 8 4 36 48 951,000 5,890.06 32 60 92% 60 92% 55 72% 55 72% ~507K 8 8 36 90 922,103 922,103 33 30-75% 80-99% 10-20% 35-75% ~500K 10 8 36 90 981,594 981.594 31 30 70% 80 96% Idle 50% 86% ~500K 12 8 36 90 1,049,290 1,049.29 29 35 76% 80-93% Idle 75-90% ~500K 16 4 42 60 1,17,0000 7,249.31 26 60 85% 60 85% 78 92% 78 92% PP - Preprocess TLO - Top Level Object RPH - Records per hour Policy Loads 1200000 1000000 ~951K ~1049K ~981K TLO/H 800000 600000 ~599K ~608K ~760K 2 DB, 2 4 DB, 2 4 DB, 4 400000 200000 ~338K 6 DB, 4 8 DB, 4 10 DB, 8 12 DB, 8 0 2 DB, 2 4 DB, 2 4 DB, 4 6 DB, 4 8 DB, 4 10 DB, 8 12 DB, 8 CPU and Thread combination Page 6

Investment Data Processing CPUs Threads Time (in TLO per hour RPH Load Size DB APP PP Commit Minutes) DB during PP DB during Commit CPU Usage during PP during Commit ~500K 2 2 12 12 371,771 371,771 79 40 92% 1-10% 20 95% 92 100% ~500K 4 2 24 24 599,387 599,387 49 40 92% 1-10% 20 95% 92 100% ~500K 4 4 24 30 699,284 699,284 42 25 92% 92 100% 1-2% 20 75% ~500K 6 4 36 72 889,998 889,998 33 30 75% 80 100% Idle 45 90% ~500K 8 4 36 90 1,048,926 1,048,926 28 30 75% 80 100% 1-2% 45 99% ~500K 8 8 36 90 1,012,757 1,012,757 29 40 80% 88 100% Idle 45 80% ~500K 10 8 36 90 1,129,614 1,129,614 26 30 75% 80 90% Idle 45 85% ~500K 12 8 36 90 1,174,798 1,174,798 25 30 70% 70 94% Idle 40 80% PP - Preprocess TLO - Top Level Object RPH - Records per hour Investment Loads 1400000 1200000 1000000 ~1048K ~1174K ~1129K TLO/H 800000 600000 ~599K ~699K ~889K 2 DB, 2 4 DB, 2 4 DB, 4 6 DB, 4 400000 ~371K 8 DB, 4 10 DB, 8 200000 12 DB, 8 0 2 DB, 2 4 DB, 2 4 DB, 4 6 DB, 4 8 DB, 4 10 DB, 8 12 DB, 8 CPU and Thread combination Page 7

5. Analysis of Results 5.1. DataXchange Enterprise: Highly Scalable Based on the results of this testing, DataXchange Enterprise is highly scalable. As the processing power of both the application and database servers increased, the number of transactions increased. Specifically, the testing illustrated that between the second, third, and fourth runs of the policy data load, the primary constraint is simply the number of processors used by the database server. Adding two additional processors to the lication Server improved the number of TLOs per hour by 9,000, and adding two more processors to the Database Server increased the number of TLOs per hour by 152,000. 5.2. Fine-tuning Prior to beginning the test runs for analysis and recording, the team worked to ensure that the infrastructure was configured correctly. E-Z Data has found that the best results are generally obtained by using the latest versions and patches provided by vendor partners. Additionally, there are specific settings for operating systems and databases that have been found to affect overall performance. 5.3. The Impact of Differing Requirements Every attempt was made to create a testing methodology and data distribution model that mimics real-life situations. However, every business is unique. Therefore, it is likely that different data, business requirements, customizations, and other types of hardware and system configurations will affect performance in different ways. E-Z Data s experienced implementation team is skilled at tuning the performance of DataXchange Enterprise to address these variations. 5.4. Lessons Learned The IBM testing process results in a number of valuable lessons learned, including: 1. The performance of DataXchange Enterprise greatly depends on the data pattern and the cleanliness of the data provided by the other systems. As part of the implementation, enterprise customers should expect to set aside six to eight weeks for fine-tuning of the application. The time required to clean existing legacy data can vary significantly. 2. Engaging E-Z Data s professional services team for the fine-tuning exercise can help significantly with the task of configuring DataXchange Enterprise for optimal performance. 3. It is recommended that the user have no more than 2 million records in the interface tables. Since these tables are designed for faster load, they do not have a high number of indexes. When the DataXchange Enterprise engine accesses these tables, the database often performs a table scan. If the 2 million record limit is exceeded, performance can be adversely affected. 4. It is recommended that stats are run on the interface tables after loading. 5. Simultaneous multithreading (SMT) should be enabled. About E-Z Data E-Z Data, Inc., established in 1986, is a leading provider of front-office systems for insurance companies, banks, broker-dealers, general agents, agents, and investment advisors, E-Z Data solutions are used by more than 100,000 advisors worldwide and over 50 leading financial services companies, including Ameriprise Financial, HSBC, MetLife, National Financial Partners, Prudential, and Securian. The company s domain expertise, coupled with mature, industry-specific business solutions, results in consistently successful customer implementations. For more information about E-Z Data s solutions, go to www.ezdata.com or call 800-777-9188. Page 8

6. endix 6.1. DataXchange Enterprise (DXE) Server In order to address the challenges described above, E-Z Data has designed and implemented the DataXchange Enterprise Server. Below is a quick overview of DataXchange Enterprise. For a more detailed discussion on the DataXchange Enterprise server, refer to the DataXchange Enterprise User Guide. Fig 1: DataXchange Enterprise Process Flow Diagram DataXchange Enterprise provides an infrastructure for data movement that includes translation of legacy or external data into the appropriate formats (Adapters), enterprise scalable programs to move and validate the data (Engines), and an infrastructure to configure, schedule, and monitor data movement processes (the Server). 6.2. Components of DataXchange Enterprise Server Extract Transform and Load (ETL) ETL is a tool used to extract data, usually from legacy systems and in some instances from consolidated databases or data warehouses. There are many options for ETL including commercial tools, bulk load utilities provided by database vendors, and homegrown tools written in C++ or Java. Once the files are extracted by ETL, they are processed by the DXE Adapter(s). DataXchange Enterprise Adapters To prepare data for processing by the engines, SmartLink Adapters translate the data into the proper format for both DataXchange and SmartOffice. E-Z Data provides Adapters for industry standard formats such as ACORD, NAILBA, and DTCC/NSCC, as well as those from standard data providers like DST and Pershing. Additionally, Adapters can be designed specifically for your internal data sources. Page 9

DataXchange Enterprise Engine At the heart of DataXchange Enterprise are the engines responsible for moving the records into SmartOffice. The solution can be used to process millions of insurance policy-related records each night, including interested parties, riders, and sub-accounts, as well as investment transactions, including the positions, transactions, and sub-accounts. The solution s multi-threaded, multi-instance architecture has been scalability tested in the labs and, more importantly, in the field. DataXchange Enterprise Console The entire process is managed from the DataXchange Enterprise console that empowers the administrator to schedule, monitor, and report on jobs. The console is able to remotely alert operators in a variety of ways if there are any unexpected events during processing. DataXchange Enterprise Server The DataXchange Server is the brains behind all of the data movement. As your console to the data processing jobs, the Server provides: An easy-to-use workbench for scheduling the various engines. Comprehensive error logs for records that have failed the validation rules. An alert system to e-mail or page the administrator when a process needs attention. Detailed reporting on engine performance and data loading statistics. Page 10