Improve performance and availability of Banking Portal with HADOOP

Transcription

1 Improve performance and availability of Banking Portal with HADOOP Our client is a leading U.S. company providing information management services in Finance Investment, and Banking. This company has a variety of different web services, applications, and databases which are serviced by different teams in geographically distributed datacenters.

2 Powering Banking Portal with Hadoop Powering Banking Portal with Hadoop Business Challenge For end-client services, client satisfaction is one of the most valuable business metrics. If you've lost your customer s satisfaction, you ve lost your customers. Maintaining customer satisfaction is a top priority for any business, but it can often be quite a tall order. There are always factors beyond our control, but luckily with prudent application of the right technologies we Web service response time (WSRT) Time to implement new useful for clients functionality Value added services Our customer experienced problems with WSRT when rolling out new features and this caused decreasing client satisfaction. As a result of our initial assessment, we delivered the customer a list of business problems, a detailed gap analysis report, and a list of potential solutions. Extreme geographic separation of development and operations teams led to miscommunication and reduced productivity, leading to degraded solution performance. A 5% performance loss in web services and analytics applications was enough to overwhelm all services, including the data center - increasing WSRT by a factor of two. All these factors led to lost profit due to release delays, both in production and in development. To ensure stability, the following recommendations were made: Control product quality and performance at all stages. Develop an automated product and server monitoring system which analyses system and application metrics as well as product health. Provide a fully automated solution. Here is a list of the most important problems, which caused decreasing product stability: Complicated solution architecture caused unpredictable effects in case of any changes to product functionality. Banking Portal with Hadoop 1 Banking Portal with Hadoop 2

3 Project Description Based on our consulting recommendations and the characteristics of their products, the client set the following Powering Banking Portal with Hadoop Automated test development environments with real production data Automated performance testing for every module during the development phase Automated performance monitoring Performance history logging with analysis features Gor Web Servers Web Apps Ambari Sensu Client Log Shipper Sensu Logstash Flapjack: Notification ElasticSearch: Search DB Kale: Anomaly Detection Engine YARN Kibana: Visualisation The concept for this solution was to provide an easy way to add availability and performance monitoring tools, from the development to production phase - from the QA team to end users. As this is a modular system, each of these modules can be easily replaced by a better one if need be. Apps Services Hadoop Cluster PROD DEV QA HDFS Hive: Query CLI Hadoop modules data flow Banking Portal with Hadoop 4 Banking Portal with Hadoop 3

4 Scaled power of Hadoop The system was designed to be a multi-layer, highly scalable platform with the ability to detect anomalies in all modules within production, development, and QA environments. Our solution included the following integrated features: Traffic forwarding This component provides the ability to forward any HTTP traffic replay in real-time in production, staging, and dev environments. This component was implemented based on the open-source tool Gor. anomaly detection component was created by Datamart LLC and is part of our Datamart Analytics Framework. Notification and Visualisation These components were built on top of the open-source tools Flapjack and Kibana. Both these frameworks provide very sophisticated API and can be integrated with almost any external modules. Flapjack provides integration with SMS gates, and Apple and Google Push Services. Metrics collection and aggregation This component uses both Ambari API to collect Hadoop cluster metrics; Sensu clients for collecting general system metrics, like CPU and Memory usage, and Logstash for collecting log files from applications. All these modules are open-source and free of cost. These powerful components can support horizontal scaling, and monitor up to 50,000 servers right out of the box. Anomaly detection This component is responsible for analysing all the data and detecting anomalies in the collected metrics. Kale, a detection reporter, uploads to the ElasticSearch database for next visualisation with the Kibana web visualisation framework. This Delivered Value Datamart LLC successfully delivered a new, ready-to-use platform monitoring system with all of the required features and capabilities. Key Benefits: Implementation of this solution allowed the size of the operations team to be reduced by half while maintaining the highest level of product availability. Banking Portal with Hadoop 5 Banking Portal with Hadoop 6

5 The overall solution performance increased by 26% in 3 months. This allowed the development team to implement new functionality twice as fast as before. New features can be tested in an environment with the same conditions as those in production, yielding realistic and reliable results. TCO was reduced. The successful results of this project led to a fruitful business partnership. Banking Portal with Hadoop 8