Assuring Global Reference Data Distribution on a Virtual Appliance

Transcription

1 White Paper Assuring Global Reference Data Distribution on a Virtual Appliance Predictive Analytics for IT in Action January 2012

2 Introduction Reference data has always been critical to the trading workflow and risk management systems. Financial services executives responsible for reference data management platforms have always been concerned about the following factors: Reducing operational risk to bank trading operations through better, more proactive management of the health and computing capacity of global reference data platforms. Providing strong management visibility into the reference data platform to ensure compliance with regulatory requirements. Providing a competitive benefit to the bank by enabling the fast and efficient delivery of consistent, high-quality reference data. Assuring the performance of global reference data systems is a problem that must be continually addressed. Demand for reduced latency is ever-present, and underlying technologies are constantly evolving, including improvements in router technology, the growing ubiquity in server virtualization, and the emergence of advanced IT analytics. Financial services organizations that take advantage of the latest technology trends will be able not only to meet the performance and compliance demands on their reference data platforms, but they will also be able to reduce the IT footprint and associated costs while providing their enterprise with a competitive edge in the capital markets. What You Will Learn in This Paper One major financial institution launched an initiative to re-engineer the part of its reference data platform responsible for the global distribution of data. The plan was to create a cost-effective virtual appliance of integrated hardware and software that could be deployed to local containers of reference data with a goal of reducing latency and increasing reliability of global reference data distribution. In this white paper you will learn about: Business requirements that led the institution to review its reference data distribution architecture. The architecture chosen by the institution for its Reference Data Virtual Appliance (RDVA). Requirements for predictive IT analytics deemed necessary to assure performance of the RDVA. Early results that make the case for further deployment of the RDVA (including Netuitive predictive IT analytics) to meet performance goals for global reference data distribution. Business Requirements for Global Distribution of Reference Data For financial institutions with global trading operations, the challenge of managing a reference data platform is not just one of complexity (i.e., the infrastructure used to aggregate and normalize data feeds) but also one of latency, usually dictated by geography. In our example, the financial institution has expanded trading operations to include centers in New York, London, Hong Kong, and Singapore while the IT infrastructure for aggregating and normalizing reference data is based in the U.K. Figure 1 illustrates the geographical distribution. Page 2

3 Figure 1. Global trading center locations for a large financial institution This geographic distribution led to the following business requirements that drove our example financial institution to review its reference data distribution architecture: With four global centers for trading operations, the reference data platform needs to perform reliably 24/7. Consumers of reference data at the trading centers are sensitive to the inherent cost of meeting their performance requirements. Each trading center expects consistent levels of service when it comes to latency and overall performance and the team wants the system to scale rapidly as trading volumes grow. Reference Data Virtual Appliance Architecture for Local Containers After an analysis of the business requirements as well as the Service Level Objectives (SLOs) for performance and latency, the architecture team for the reference data platform decided that the solution was to rollout infrastructure to establish local containers for the data in close proximity to each trading center. At the same time, the team was also mindful of the cost vs. performance and scalability concerns in designing the IT infrastructure to be installed at each local container. The resulting design was called the RDVA. Each RDVA is designed to maintain access to the Reference Data Master via a wide area network (WAN) connection. Each RDVA unit was standardized with the following physical, virtual, and hardware components so that the infrastructure could be scaled and managed easily (see Figure 2 for an illustration of this list): 1. Low-latency routers to handle data feeds from the Reference Data Master. In this case, a hardware network appliance, the Solace 3260 Messaging Router, was selected. 2. A software solution for rapid data replication and distribution. In this case, Oracle Golden Gate software was selected. 3. A high-performing database solution to serve as the local reference data repository. In this case, Oracle 11g was selected. Page 3

4 4. A Service Oriented Architecture (SOA) middleware platform for handling request/reply (pull) and pub/sub (push) requests for reference data from various applications. In this case, TIBCO ActiveMatrix BusinessWorks (AMX BW) was selected. 5. A virtualization platform to manage host servers and virtual machines on which TIBCO instances will be deployed. VMware was selected as the virtualization platform allowing flexibility in server configuration and high end host server specifications were used to minimize any concerns about computing capacity or performance. 6. A holistic analytics and monitoring solution to correlate performance across the RDVA architecture. Alreadylicensed monitoring tools will be used with Netuitive software as a predictive analytics overlay. The Netuitive software will be deployed on a virtualized server. Assuring Performance of the Reference Data Virtual Appliance By deploying TIBCO AMX BW on a virtualized platform, the architecture team hopes to take advantage of the inherent configuration flexibility of virtual machines. Different sized virtual servers can be configured to handle the varying loads from different applications requesting reference data. See Figure 2 for an illustration of the RDVA component and monitoring architecture. Figure 2. Architecture and performance monitoring solution for the RDVA Page 4

5 The diverse RDVA architecture posed a challenge to monitoring the performance of the reference data distribution service in the following ways: Individual tools exist to monitor the individual components of the RDVA, but it is difficult to understand the holistic performance of the platform. With different tools and siloed views, diagnosing performance problems can be extremely difficult or impossible. The new architecture and unprecedented scale and reach of the reference data service meant that the architecture team had no idea what to expect when trying to understand normal performance or global latency. The team had limited experience deploying critical services on a virtualization platform, so something had to be done to minimize the added operational risk this introduces. It became obvious to the architecture team that they needed a new, holistic performance monitoring approach to ensure the performance of the RDVA platform one that perhaps leveraged the latest in IT analytics technology. Netuitive software was selected as the best solution using the following evaluation criteria: The software must leverage existing monitoring tools to collect performance metrics without adding any significant load to the RDVA components and without duplicating metric data storage. The software must be capable of collecting and analyzing performance metric data for physical as well as virtual components of the RDVA, including the router (Solace), physical servers (UNIX), virtual servers (VMware), databases (Oracle) and the SOA data service platform (TIBCO AMX BW). The software must be able to create and visualize service models and topologies to bring together multiple components and metrics in the RDVA architecture. The software should have the ability to discover and calculate statistical correlations between the performance metrics for each component of the RDVA, thereby allowing a clearer, holistic understanding of how the RDVA is performing. The software should be able to generate alarms at a component and system level based on deviation for expected behavior or performance. The software should allow users to set SLOs that override behavior-based alarming where appropriate. Based on their experience with existing monitoring approaches, the team realized that traditional holistic monitoring of the RDVA could not be accomplished simply with a unified dashboard dependent on human analysis. This was simply not scalable, and the results were unreliable. Event correlation approaches also have not worked. This approach is dependent on human expertise to analyze and understand the correlated behavior of hundreds of performance metrics for components in the RDVA. This expertise simply did not exist, nor was it scalable. Instead, the team chose Netuitive s predictive analytics because of Netuitive s broad portfolio of integrations to existing monitoring tools and because of Netuitive s analytics technology. Netuitive s analytics engine is powered by a unique Behavior Learning Engine that can automatically learn how all the different performance metrics from RDVA components should behave in correlated fashion, even as performance loads vary by time of day or day of the week. The same technology also correlates the behavior between the components of the RDVA for a holistic performance view of the reference data service. Furthermore, the architecture team was confident in the technology because it is based on sound fundamental statistical methods that have been proven in other areas, including computerized trading and actuarial science. Page 5

6 Deploying Netuitive Predictive Analytics for the RDVA To implement the total RDVA solution, the architecture team leveraged standard Netuitive integrations to IBM Tivoli Management and VMware to collect performance metrics on the Oracle database and the operating systems of physical and virtual servers in the RDVA. Netuitive was engaged to develop integrations to the Solace router and to TIBCO AMX BW. Component and Service Health Scores Figure 3 shows an at-a-glance view of the entire component estate of the RDVA. Each block in the graphical interface represents a physical, virtual, or software component of the platform organized into component types by folder. The blocks turn from green, to yellow, to orange, to red depending on the component or service health. A service or component health score is an aggregate measure of how normal the behavior of each component or service is for the current time of day or day of the week. Netuitive s definition of normal includes comparison to historical behavior of components and services as well as a unique understanding of the current context of behavior. For example, if services are consuming resources unexpectedly during a time period but Netuitive sees a correlated rise in the number of business transactions, this could be interpreted as normal behavior. By drilling down into each folder of component type, users get more detail about component health. Figure 4 shows a drill down into the specific health score (a number from 0 to 100) of RDVA services modeled in Netuitive. The first score is a rollup of the overall performance and health of the entire RDVA unit (virtual appliance), and other minor services, including the performance of the reference data request/response service, which is a software subservice of the entire unit. Virtual Appliance Test Service Request Response Figure 3. Component and service health at-a-glance Figure 4. Service health scores Page 6

7 Component and Service Relationships Figure 5 illustrates that the overall RDVA health is a composite of the health of its components: namely, the Solace 360 router, the reference data request/response service, and the supporting TIBCO and VMware platforms. In each case, Netuitive is indicating that all components and the overall RDVA are performing normally (as expected) at this time. RDVA - (Reference Data Virtual Appliance) Figure 5. Health and topology of the RDVA and underlying components Component and Service Level Behavior Analysis Netuitive s definition of normal is that the most relevant performance metrics for each RDVA component are operating within operational baselines calculated by Netuitive. Not only does Netuitive calculate historical baselines (the green band in Figure 6), but also contextual baselines (the blue band in Figure 6). Contextual baselines determine where a metric is predicted to be based on other correlated metrics. Contextual baselines are unique to Netuitive and yield much more accurate alerting to anomalous behavior in a component or service. Certain performance metrics from two different components in the RDVA may behave in opposite or like fashion during normal operation. Understanding these correlations will help application and IT infrastructure support staff with diagnostics when there are performance issues. As stipulated in the evaluation criteria, Netuitive can automatically calculate correlation coefficients between metrics from multiple components within a service model, enabling a contextual understanding of the whole service behavior. This is illustrated in Figure 7, where Netuitive is correlating the behavior between a specific virtual machine and a service in TIBCO. Figure 6. Historical and contextual baselines for two RDVA metrics Figure 7. Correlation detected between a VM and a TIBCO service Page 7

8 Anomaly Detection and Alerting In addition to helping users to better understand the holistic performance of the RDVA, the Netuitive Behavior Learning Engine can alert users to anomalous behavior, which may have a future impact on the performance of the reference data service. Two examples of this are illustrated below. The first example appears in Figure 8. In this example, Netuitive is alerting users that performance metrics are not being generated on the request/response service, thereby introducing an operational risk due to lack of service management. The second example appears in Figure 9. This example illustrates a performance anomaly in a virtual machine on the RDVA, indicating that eight performance metrics have deviated for expected normal behavior. In both cases, Netuitive is providing an actionable warning for IT support staff to analyze the problem before it has persistent impact on the reference data service. Figure 8. Netuitive alerting to missing data Figure 9. Netuitive alerts to anomalous behavior in a VM Root Cause Isolation When performance anomalies occur, the service and component dependencies modeled and discovered in Netuitive enable automated problem isolation via automated behavior correlation, which helps in diagnosing incidents. The examples in Figures 10 and 11 illustrate anomalous behavior in one of the virtual machines (lcamm21605) happening concurrently with missing data alerts. A timeline view clearly shows the loss of data due to abnormal behavior of the TIBCO server virtual machine, which was restarted thereby stopping the TIBCO services. Figure 10. Anomaly in a TIBCO virtual machine Figure 11. Data loss due to a virtual machine reboot Page 8

9 Service Level Objectives Netuitive will only alert users after a significant number correlated metric anomalies have occurred. However, regulatory or business service level agreements may dictate otherwise. The dialog box appearing in Figure 12 below illustrates a user setting up an SLO criteria to alert as soon as the first metric deviation of any kind is detected in an important web service in the TIBCO server. Figure 12. Setting an SLO to override Netuitive performance alarms All the implementation examples listed here illustrate how Netuitive was able to meet the stringent evaluation criteria for an analytics platform for the RDVA. Page 9

10 Results Value Delivered The financial institution in our white paper believes that their plan to deploy RDVA units to regional local containers will greatly improve the performance and reliability of the global distribution of Master Reference Data. And while the RDVA architecture offers a scalable and flexible way to improve reference data service performance globally, the new solution also introduces risk. By deploying Netuitive in the RDVA solution, the institution feels that the benefits of Netuitive s predictive analytics are to: Reduce risk to bank trading operations by providing proactive performance management of the reference data platform and delivering real time visibility into health, workload, and operational headroom of the global reference data platforms. Provide evidence of strong management visibility into reference data platforms used by the bank to ensure compliance with regulatory requirements. Provide a competitive benefit to the bank by enabling the fast and efficient delivery of consistent, high quality reference data. As a next step, the bank plans to complete the testing and roll-out of the RDVA units globally. Following that, the bank is considering integrating performance metrics from Geneos (an ITRS Group product) for better end-to-end performance management of the entire trading workflow. Conclusion Financial institutions continue to leverage new technologies to improve the cost-effectiveness and performance capabilities of their reference data platforms. This includes new router and database technologies, as well as server virtualization technologies. But the introduction of all these new technologies in a new solution architecture can increase operational risk, even as the technologies promise to reduce reference data distribution latency. It is simply not possible to ensure the performance of these composite solutions using a plethora of monitoring tools with myopic views for performance metrics. Netuitive provides a comprehensive analytics layer across these monitoring tools to collect, correlate, and analyze metrics across physical and virtual components of reference data platforms. A unified performance dashboard also provides holistic executive insight into the performance of services. Finally, Netuitive s sophisticated analytics alert users to impending performance problems so that they can be proactively addressed by support staff before trading operations are impacted. Page 10

11 Netuitive Predictive Analytics for IT Netuitive, Inc Sunrise Valley Drive Reston, VA USA Toll Free: Main: Netuitive Europe Ltd 33 Throgmorton Street London EC2N 2BR United Kingdom Main: Web: Proactive. Predictive. Preventative. Netuitive provides predictive analytics software for IT. Netuitive replaces human guesswork with automated mathematics and analysis to forecast, identify, and resolve IT performance issues before they impact quality of service. Hundreds of customers, including eight of the 10 largest banks, rely on Netuitive to proactively manage the performance of their critical applications and underlying IT infrastructures physical, virtual, and cloud. Industry recognition includes the 2011 CTO Award for Innovation from Morgan Stanley, the 2011 CODiE Award for Best Systems Management Solution, the 2010 EMA Award for Best Analytics, and Best of VMworld Awards in 2007, 2009 and For more information, visit Contact Netuitive to learn more about our solutions for proactive IT performance management, or to schedule an on-site demonstration. Netuitive, Inc., No material in this document can be reproduced without the expressed written consent of Netuitive, Inc. Page 11 Platforms Netuitive for Reference Data