Migrating a real-time transaction evaluation and scoring solution to HP NonStop platform

Technical white paper Migrating a real-time transaction evaluation and scoring solution to HP NonStop platform Table of contents Introduction... 2 Evaluation service... 2 Scoring service... 2 Migration overview... 2 Database migration... 3 Table keys and partitioning... 4 Database access... 4 Application migration... 4 Evaluation service... 4 Spring... 4 Inter-process communications... 5 Scoring service... 5 Apache Tomcat-based user interface application... 5 Application deployment using Pathway... 5 General software vendor architecture and the Pathway environment... 6 Evaluation service... 9 Scoring service... 11 NSJSP Based on the Apache Tomcat Servlet container... 12 Conclusion... 15 Reference information... 15

Introduction A large corporate customer was looking for a real-time fraud detection application. They wanted the software to be flexible and to be able to perform extremely well, even under high transaction volumes. The software had to be scalable so that it could grow with their business. It had to be cost-effective, manageable, and always available. The customer found an application that met their needs, but it wasn t NonStop. This paper describes how HP and the software vendor responded to the customer s demand/request to port their real-time fraud detection applications to the HP NonStop platform. The migration of a software vendor s real-time transaction evaluation and scoring applications from Oracle to HP NonStop demonstrates the ease and advantages of re-hosting a modern application based on some of the leading Open Source Frameworks. The NonStop platform supports many of the most popular frameworks and provides a compelling application development environment using Open Source tools like Eclipse. This software vendor provides payments processing and fraud prevention services and products around the world. The fraud prevention product suite includes applications used to detect fraud in real-time (in the transaction path) and near real-time (shortly after the transaction was processed). This white paper discusses the migration of the real-time fraud detection (evaluation service) component and the scoring service of their application suite, which differentiates this solution from others in the marketplace. Evaluation service The evaluation service application provides fraud detection as a service to other applications. It typically sits in the middle of ATM and/or POS authorization processing, receiving transactions in the form of an industry-standard message and responding to the calling application with an industry-standard message. The evaluation service, when directed by configurable business rules, calls the scoring service for required services. The evaluation service application was built using a Plain Old Java Object (POJO) approach, and various Open Source Frameworks, including Spring Framework (Spring) and Hibernate that are available on the NonStop platform. These frameworks and the flexible programming environment on the NonStop facilitated a smooth application migration. The Open Source Frameworks were augmented with the HP JToolkit for NonStop servers. This allowed the evaluation service to function in the HP NonStop Pathway environment as a Pathway server class. Spring and Hibernate combine to eliminate vendor-specific database access issues. Scoring service The scoring service application was built using the C language and embedded SQL. The application was well-structured for change but required a full rewrite of the database access portion of the code to replace Oracle OCI with embedded SQL/MX statements. Migration overview This migration was done as a proof of concept (POC) in the context of eventual deployment on HP Integrity NonStop BladeSystem at a customer site. The customer provided service level expectations for the maximum allowable response time and an expected transaction throughput rate. There was an implicit assumption that the application could be scaled out to provide even greater throughput with reduced response time. Much of the POC activity was geared toward meeting these goals. The migration occurred in three distinct phases, beginning with a pre-poc analysis of the code and database. This was a week-long effort that involved several contributors. The code and the Oracle database schema and tables were evaluated, primarily by manual methods. This revealed differences in Oracle and NonStop DDL, the use of triggers, and a cursory review of some representative PL/SQL code. The application review looked for the general architecture of the various applications and for SQL statements buried in the code. A directional statement with a straw man approach was produced and delivered to both the software vendor and HP management teams so that a joint go forward business plan could be developed. The beta version of the Ispirer SQLWays tool was used to assist with the database evaluation. As a byproduct, feature and functional gaps in the tool were identified by its use in this real life case. With this feedback, Ispirer enhanced the tool, and the current version of SQLWays incorporates fixes and improvements for DDL conversion, trigger features, PL/SQL functionality, and others. The second phase of the migration was the POC to evaluate the feasibility of running the code as-is with minor code changes and a straightforward database migration. This was important to demonstrate the openness of both the software vendor s applications and the NonStop platform. 2

As the POC evolved, the project team identified code and database changes that would improve performance either generally or on the NonStop specifically (primarily the addition of indexes and re-partitioning of tables). Small code changes were applied to the applications so that NonStop technology could be better exploited. Many of these were implemented in a post-poc phase that also prepared the applications for pilot deployment in the customer s production environment. Application performance was continuously monitored as changes were staged and deployed. Database migration The initial review of the Oracle version of software vendor s database indicated no showstoppers and very few unexpected uses of Oracle database technologies. The DDL was reasonably straightforward and made use of only a small number of Oracle data types. The chart below (Table 1) maps Oracle data types to the NonStop SQL/MX equivalents used for this project. Table 1: Oracle to SQL/MX data types Oracle data type Data characteristic NonStop SQL/MX data type Char Fixed length character data Char VARCHAR2 Variable length character data VARCHAR Date Date and time to second Timestamp (0) Date Date and time to microseconds Timestamp (6) or Timestamp Number (less than 5 digits) 2 byte integer value Smallint Number (less than 10 digits) 4 byte integer value Integer Number (less than 19 digits) 8 byte integer value Largeint Integer Defaults to number (38) Numeric (38) Number(p,s) Number (>18 and <129 digits) BLOB P = Precision S = Scale Precision = number of digits Scale = 0 BLOB content can be saved in a VARCHAR column if size for data, length indicator, and optional null terminator is <= available space in the row Numeric(p,s) Numeric (number of digits) VARCHAR The initial review did reveal the use of column names that are reserved words for NonStop SQL. These column names were action and value. Although NonStop SQL provides the option of quoting the column names so that the database engine no longer considers them to be reserved words, this would require changes to the application code wherever the column names were referenced. Because the application had to be changed in any case, the HP and software vendor team chose more meaningful column names and changed the code and the database. The initial review also identified triggers within the Oracle database. Ultimately, these were not used in the NonStop SQL deployment, but new triggers were added during the POC replacing some programmatic functions. The Oracle PL/SQL code was re-written and re-hosted as methods (functions) within the software vendor s near-real-time (NRT) application (not described in this document or included in the new application code created for the NonStop implementation). 3

Table keys and partitioning Identification of appropriate primary key, storage and partitioning columns is crucial to optimizing performance on the NonStop platform. Proper partitioning of database tables must accommodate even distribution of data volume and access across all partitions and computing resources. Two partitioning options are available: range and hash partitioning. Using range partitioning with natural keys can cause uneven data volume distribution in some cases. As is common among Oracle database deployments, surrogate keys are heavily used in the software vendor s database, and the Oracle Sequence Number Generator is used to produce unique identifier (UID) values for surrogate keys. While this may be acceptable for Oracle deployments, it did not meet this project s requirement for a UID that provides uniqueness, sequentially ordered data, and flexibility for partitioning. Code to generate a NonStop UID that met user requirements was developed in Java and packaged in a library that is callable by various application objects. Database access The Java-based evaluation service application uses JDBC (Type 2 driver) and Hibernate to access database tables. Appropriate property files (jdbc.properties and hibernate.properties) are used to configure the respective data access products. Many (or most) proprietary vendor database deployment considerations are hidden by these products. Hibernate also requires a dialect file that allows the standard Hibernate framework to function correctly with each vendor database product. The dialect file, hibernate3sqlmx.jar, is provided for use with NonStop SQL/MX. JDBC features, like connection pooling and statement caching, required further consideration. These are supported by the JDBC driver and application level libraries that are discussed later in this document (under Application Migration: Evaluation Service). Application migration This migration involved Java and C applications, which will be discussed separately here. This section also includes discussion of a Web-based user interface. Deployment of the evaluation service and scoring service applications using Pathway is covered as a single topic in the following section. Evaluation service Evaluation service is a Java application that consists of several branches of business logic and depends on access to several tables, including a transaction log. Key to the ease of migration were the structure of the application and the use of Open Source Frameworks, primarily Spring and Hibernate. The Model-View-Controller (MVC) design pattern was used to provide consistent structuring and layering for all code components. This method of code organization requires decoupling of business code and data access code. Thus, it was a relatively simple and straightforward task to use a NonStop data source in place of an Oracle data source. Minor changes to the jdbc.properties and hibernate.properties files were required. Two libraries (jdbcmx.jar and hibernate.jar) were added to the application lib directory to support the NonStop deployment. A relatively new NonStop product, Module File Cache, was configured for use by the evaluation service application. It is enabled through modification of the jdbc.properties file. Hibernate capabilities were augmented using two additional libraries (c3p0 and EHCache). The application manages a connection pool facilitated by c3p0 and a statement cache supported by EHCache. Spring Spring manages transaction boundaries (transaction unit of work) using declarative transaction attributes. These are implemented through the use of annotations and support the following: Propagation behavior Isolation level Read-only access Transaction timeout Rollback rules (no-rollback-for and rollback-for) Spring annotations alone did not provide the desired level and granularity of transaction management. Transaction boundaries were delineated at specific points in the code by introducing a pointcut where we wanted to apply aspect advice. 4

Inter-process communications Previous versions of evaluation service used TCP/IP sockets to communicate with the front-end switch application and the Scoring Service. While this is a workable model on the NonStop platform, it does not exploit the platform s ability to scale out simply and efficiently. Use of sockets I/O also imposes greater latency compared to message passing, the standard practice on the NonStop platform. Scoring service The scoring service is used by the evaluation service application to refine its evaluation of the probability of fraud for the current transaction. It is written in the C language and compiled successfully using both the Eclipse-NSDEE-PC-based C language cross compiler and the NonStop host-based compiler. Several previously used libraries were not needed for deployment on the NonStop platform and a few libraries were added for the same reason. Previous releases of the scoring service application supported the Oracle and IBM database engines, each accessed through their respective call level APIs (like Oracle s OCI). The NonStop scoring service deployment uses embedded SQL statements to provide a static (predictable) and better performing environment in which SQL statements are precompiled and data access is optimized. The scoring service application is deployed as a context-free and single-threaded service, making it a perfect candidate for deployment as a NonStop Pathway server class. The Pathway model has additional benefits: the scoring service could be run under multiple server classes, each with its own priority level and deployment characteristics. Apache Tomcat-based user interface application The software vendor s product suite includes some Web-based user interface applications developed for the Apache Tomcat environment. The real-time Web console, for example, is used to maintain configuration and business rules tables. This application was initially tested using the NonStop NSJSP product that works in conjunction with the HP itp WebServer product. NSJSP is a deep port of Apache Tomcat 6 that has been optimized for the NonStop platform. The latest version is compatible with Apache Tomcat 7. The real-time Web console also uses Open Source Frameworks and the MVC design pattern. Database access occurs via Hibernate and JDBC, initially with the Type 2 driver. During the course of the POC, the customer decided to deploy the application in a Tomcat container running on a Linux server. This application re-hosting is easily accomplished as the NonStop resident database is remotely accessed using the SQL/MX JDBC Type 4 driver. Application deployment using Pathway NonStop deployments typically run in Pathway environments that use TS/MP and related products as this is the easiest way to exploit the platform. TS/MP supports ease of development and manageability and the fundamental strengths and benefits of NonStop systems: data integrity, fault tolerance, high performance at low cost, system security, scalability, and distributed processing. Availability, scalability, manageability, and the ability to meet or exceed performance objectives were important for this customer. The previously proposed deployment offered less availability and scalability and the potential of failing to meet performance goals all built on a more complex and less manageable group of loosely coupled servers. Use of modern design principles allowed the software vendor s Java and C applications to slip neatly into Pathway environments on the NonStop Server and to reap NonStop benefits. 5

General software vendor architecture and the Pathway environment Figure 1 shows the general software vendor application architecture as originally deployed. TCP/IP with socket I/O is used for communication between the various platforms and applications. Figure 1: Original software vendor deployment NonStop Linux Linux Switch TCP/IP Evaluation Service () TCP/IP Scoring Service () Messages originate with the switch application, which resides on a NonStop system, and are transmitted to the evaluation service (evaluate) application, originally deployed on a Linux system. The evaluate application sends a specially formatted message to the scoring service (score) application on another Linux server. The evaluate application combines the contents of the score s reply with the results of its own processing and replies to the switch application. Figure 2 shows a simplified, shorthand version of Figure 1. It will be a building block of future diagrams. Figure 2: Simplified view of original architecture IP-address:port IP-address:port Switch This deployment works best when each of the applications is deployed as a single instance and a specific IP address and port combination is associated with paired instances of the evaluate and score applications. The flexible software vendor architecture allows scaling out by using additional Linux servers to host more copies of the evaluate and score applications as the workload increases. Each application instance requires a unique IP address and port number. Data distributors are required to scale out the deployment of the sockets-based applications and provide some degree of load balancing across all computing resources. To make this work, each distributor maintains a list of port numbers that are associated with the target application instances. As incoming transactions are accepted, the data distributor decides which resource to utilize and routes the transaction to that resource using the appropriate IP-address and port number. While this permits better transaction distribution over the various instances deployed at each level of the architecture, it is a complex environment to manage, as shown in Figure 3. 6

Figure 3: Original scale out architecture Switch 1000 Data distributor 1001 1100 Data distributor 1101 for evaluate instance 1 for score instance 1 1002 instance 2 1102 instance 2 1003 instance 3 1103 instance 3 1099 instance X 1199 instance X Although many details have been hidden, figure 3 shows that unique port numbers must be assigned to each server/service instance. This indicates probable operational complexity, because each of these connections must be maintained while the system of systems is up and running, which, for this kind of application is likely to be 24 hours a day x 7 days per week x 365 days per year. The use of data distributors, while beneficial for improved scalability, adversely affects the availability characteristics of this architecture. Failure of an instance of either data distributor prevents any transaction from being completed. Introduction of redundant data distributors helps to relieve some of the availability issues. This assumes that the switch and evaluate applications have built in mechanisms that can be used to retry failed transactions. This also assumes that each application can dynamically reroute transactions to the appropriate available resource. All of this messiness real and potential can be eliminated with a Pathway deployment where each application communicates with other applications using the standard NonStop message interface, the $receive file, used for inter-process messaging on NonStop systems. The Pathsend API masks the low-level nuances associated with direct file system I/O. Figure 4 illustrates this. Figure 4: Simple NonStop deployment architecture NonStop NonStop Switch Pathsend Evaluation Service () Pathsend Scoring Service () In this example, the evaluation service (evaluate) and scoring service (score) applications are deployed as Pathway server classes on the same NonStop system. However, they could have been simply and transparently deployed on two different NonStop systems. 7

The server classes (evaluate and score in the diagram) are configured to start N number of static server instances in assigned CPUs at an assigned relative priority level. The actual number of processes that service requests at each level of the architecture is independently managed and does not rely on a data distributor level. Elimination of the data distributor level also reduces latency. Underlying sub-system components manage the complexity of keeping communication channels open and available. These same sub-system components attempt to balance the transaction load across the configured server process instances of the server class (resource pool). Figure 5 shows the Pathway version of figure 3. Figure 5: NonStop deployment showing Server class pools Switch Pathsend Pathway server class Pathsend Pathway server class instance 1 instance 1 instance 2 instance 2 instance 3 instance 3 instance N instance N Under Pathway, the switch application views the evaluation service as a pool of generic resources, that is, the evaluate server class. The switch sends a transaction request message to the generic evaluate server class, and the underlying system software determines which instance should service the request. Likewise, there is no coupling of paired instances of the evaluation service and the scoring service: the evaluation service application, running as an instance in the evaluate server class, sends its requests to the generic score server class. Underlying system software determines which score instance should service the request and it ensures that reply messages are returned to the source of the request. Server class pools are configured to be as large as required to match workload demands. NonStop sub-system components transparently assign and utilize server class process instances in a balanced fashion. This simple but elegant architecture proves to be beneficial for both application scalability and availability. Experience has shown that complex environments are inherently less reliable and less available than simple environments. 8

Evaluation service Prior to the port to NonStop, the Java-based evaluation service application used socket I/O to communicate with the front-end switch and scoring service applications. The NonStop deployment uses direct message I/O through the underlying NonStop kernel message and file systems. For Java-based applications, this is facilitated by use of the JToolkit product. It includes library functions that support JPathway (capability to read messages from $receive) and JPathsend (capability to send messages via $receive to a Pathway server class). Figure 6 shows how the evaluation service application was structured to handle socket I/O. Figure 6: Evaluation service (evaluate) using socket I/O to communicate with remote applications Switch TCP/IP TCP/IP Socket I/O handler Socket I/O handler Spring Framework Core libraries Business logic Spring DAO implementation Hibernate JDBC Oracle database 9

Figure 7 shows the NonStop version of the application, which uses the NonStop message and file systems ($receive) for inter-process communication. Figure 7: Evaluation service (evaluate) using NonStop message handling to communicate with remote applications Switch Pathsend Pathsend JPathway (from JToolkit) Spring Framework Core libraries JPathsend (from JToolkit) Business logic Spring DAO implementation Hibernate JDBC NonStop SQL database NonStop functionality was introduced with minimal impact to any of the core application functionality. Only the socket I/O handling methods needed to be replaced by the JToolkit libraries and methods. 10

Scoring service The scoring service application acts as a service to the evaluation service application. It is written in C, not Java. In the previous version, the evaluation service and scoring service applications communicated via TCP/IP using socket I/O, as shown in figure 8. Figure 8: Scoring service (score) using socket I/O to communicate with evaluation service (evaluate) TCP/IP Socket I/O handler Business logic Oracle OCI statements Oracle table The NonStop version uses the NonStop message and file systems ($receive) for communication between NonStop processes. Because the scoring service application is written in C and uses standard library functions, I/O for the $receive file uses regular file system calls. This is shown in figure 9. 11

Figure 9: Scoring service (score) using NonStop message handling to communicate with evaluation service (evaluate) Pathsend Read and write $receive Business logic Embedded SQL statements NonStop SQL table The overall structure of the scoring service code remained intact with minor changes for message handling and database I/O. The business logic was not altered. NSJSP Based on the Apache Tomcat Servlet container The real-time Web console application was originally tested with NonStop NSJSP and itp WebServer. Ultimately, the customer chose to deploy the application in a Tomcat container running on a separate system. Deploying a Web application to the NSJSP Servlet container on the NonStop Server is the same as deploying the Web application to the Apache Tomcat Catalina container on a Linux server. The difference is application scalability because the NSJSP container is deployed as a TS/MP (Pathway) server class. 12

Figure 10 shows how NSJSP is deployed under itp WebServer. Figure 10: NSJSP deployment under itp WebServer 1 Network TCP/IP v6 CPU 0 CPU 1 CPU 2 CPU 3 Szweb HTTPD HTTPD HTTPD HTTPD itp Secure WebServer Pathsend or File system AP NSJSP NSJSP NSJSP NSJSP NSJSP Installation The following manual excerpts explicate figure 10: NSJSP 7.0 is an enhanced implementation of the Apache Tomcat 7.0.10 release. NSJSP is designed to be installed in an itp Secure WebServer environment and to use the WebServer to handle the HTTP protocol. A NonStop connector was developed for NSJSP to enable communication between the itp Secure WebServer and NSJSP, using Inter-process Communication (IPC). NSJSP processes are also designed to run in server classes, which are defined as part of an itp Secure WebServer TS/MP configuration. 2 NSJSP architecture shows the following: An itp Secure WebServer installation in a TS/MP environment with a PATHMON, called $zweb. The itp Secure WebServer Server Class has multiple HTTPD processes distributed across four CPUs. A single NSJSP installation in this itp Secure WebServer environment. The NSJSP installation also comprises multiple NSJSP processes distributed across the four CPUs. The HTTPD processes communicate with NSJSP processes using either Pathsend API or the file system API. 3 1 NonStop Servlets for JavaServer Pages (NSJSP) 7.0 system administrator s guide p. 1 17 2 NonStop Servlets for JavaServer Pages (NSJSP) 7.0 system administrator s guide p. 1 2 3 NonStop Servlets for JavaServer Pages (NSJSP) 7.0 system administrator s guide p. 1 17 13

Figure 11 shows message flow for a user request. Figure 11: Virtual hosting in a sample itp Secure WebServer environment 4 This architecture makes it easy to exploit existing services that are already deployed in Pathway server classes. The web application functions as the client of the desired service. An appropriate request message can be sent to the service using Pathsend functions. Figure 12 illustrates how a pathway hosted service can be accessed from a web application. Figure 12: Accessing a Pathway based service from a Web application 4 NonStop Servlets for JavaServer Pages (NSJSP) 7.0 system administrator s guide p. 3 83 14

Conclusion Customers derive many benefits through migration of applications to NonStop. Typically these benefits are inherited through use of the NonStop Pathway environment. Plain Old Java Objects are well supported by the NonStop platform, especially when deployed as Pathway server classes. The NonStop Server supports popular Open Source Frameworks that can ease the migration effort. Key frameworks include Spring and Hibernate. Data access through an industry-standard framework like Hibernate or through use of the JDBC API masks differences between vendor database products. Database migration can be a straightforward and easy task if RDBMS vendor-specific functionality was avoided. Open source and third-party tools are available to assist with the migration effort should they be required. The migration of this software vendor s applications is one more proof point that enterprise class applications can find a new home and flourish on the HP NonStop platform. Additional information is available for the actual customer deployment of the software vendor s applications on NonStop. Reference information Detailed information about the references mentioned below is available at NonStop Technical Library (NTL) hp.com/go/nonstop-docs HP Integrity NonStop J-Series C/C++ Programmer s Guide for NonStop Systems Guardian Procedure Calls Reference Manual itp Secure WebServer System Administrator s Guide JDBC Type 2 Driver Programmer s Reference for SQL/MX Release 3.1 JDBC Type 4 Driver Programmer s Reference for SQL/MX Release 3.1 JToolkit for Java API Reference Pages NonStop Development Environment for Eclipse (NSDEE) Pointer NonStop Development Environment for Eclipse Debugging Supplement NonStop Development Environment for Eclipse User Guide NonStop Server for Java 6.0 Programmer s Reference NonStop Server for Java 6.0 Tools Reference Pages NonStop Servlets for JavaServer Pages (NSJSP) 7.0 System Administrator s Guide NonStop Servlets for JavaServer Pages (NSJSP) Configuration White Paper Open Source Java Frameworks on NonStop User s Guide (Spring 3.0.2, Hibernate 3.5.1, My Faces 2.0.2, Axis2/Java 1.5.2) SQL/MX 3.1 Programming Manual for C and COBOL SQL/MX 3.1 Reference Manual TS/MP Pathsend and Server Programming Manual Get connected hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Java and Oracle are registered trademarks of Oracle and/or its affiliates. 4AA4-0366ENW, Created May 2012 15