How a global bank is overcoming technical, business and regulatory barriers to use Hadoop for mission-critical applications

Transcription

1

2 Case study: How a global bank is overcoming technical, business and regulatory barriers to use Hadoop for mission-critical applications Background The bank operates on a global scale, with widely distributed operations across North America, Latin America, Europe, Africa and the Asia-Pacific region. The bank has over 225,000 employees worldwide and just over $2.5 trillion in total assets. Financial offerings include consumer and commercial loans, credit cards, mortgage and home equity loans, and brokerage and wealth management services. Customers range from consumers and small businesses to Global 1000 corporations and large government agencies. The bank has thousands of software applications spanning retail and commercial banking, investment management, fraud prevention, risk analysis, and regulatory compliance. It is subject to a broad range of regulations and standards in areas such as privacy of customer information, money-laundering, marketing practices, reserves and financial exposure, and financial reporting. Initial Hadoop projects allowed the bank to achieve significant cost savings in comparison to traditional data warehouse solutions. However, some IT executives were concerned that Hadoop was not as mature as established technologies. They pointed to specific shortcomings that would make regulatory compliance difficult, limiting the number of applications the bank could use it for, as well as limiting the infrastructure cost savings the bank was hoping to achieve. Challenges The bank realized that by moving applications to a Hadoop infrastructure it could: Strengthen real-time decision making by processing high volumes of data in short time windows. Gain a better understanding of its customers and markets by collecting and analyzing unstructured data, such as clickstream data, text from forms and s, server logs, and posts from social media tools, alongside structured account data. Improve analysis by cost-effectively storing data for much longer periods. Reduce costs by running major data processing tasks on commodity hardware. However, several characteristics of Hadoop posed problems for the bank s IT staff. Among the most important issues: Backup, reliability and data consistency across clusters and locations is problematic with standard Hadoop. Tools normally used for backing up Hadoop clusters are based on DistCp, which is primarily designed for copying files, not consistent replication of an entire cluster for backup. DistCp can consume significant cluster resources, preventing other applications from running efficiently. //01

3 This meant that it wouldn t be practical to run DistCp for backup more than once a day. As a result, the backup would be at least 24 hours out of date with changes on the primary Hadoop cluster. Load balancers can help, but require constant administrator attention and are not completely reliable. Load balancers can run out of buffer space when the WAN is down between locations, resulting in potentially unrecoverable data loss. Complexity increases dramatically as more data centers are added. Inevitably, data on primary and backup clusters diverges over time when DistCp and solutions based on it are used. This is unacceptable from a regulatory perspective. When divergence occurs, resynchronization of the backup cluster with the primary requires manual effort and can make the cluster unusable until full recovery is achieved. During its Hadoop pilot, the bank also noticed performance issues with applications that involved large numbers of small files generated in short timeframes. Hadoop s single active NameNode architecture became a bottleneck preventing other applications from running on the cluster while these applications were running. Guaranteeing optimum performance is difficult with mixed workloads. The bank s IT group planned to deploy several critical real-time analytics applications on high-performance systems with extra RAM to support in-memory computing. However, Hadoop does not have a mechanism to zone application processing within a cluster; all applications are distributed equally across the same infrastructure in each cluster. This confronted the bank with three undesirable alternatives: Put all Hadoop applications on high-end hardware, which would break the budget allocated for their Hadoop deployment. Spread all Hadoop applications across a combination of high performance and standard hardware, which would produce sub-optimal performance for the critical applications. Create separate Hadoop clusters with different hardware for each application type, which would lead to redundancy, waste, inflexibility, and higher management costs. Privacy laws complicate the data lake. In several European Union nations, Argentina, Saudi Arabia and other countries, data protection and privacy regulations prevent or severely restrict the transmission of personal information outside of national borders. Hadoop does not provide a mechanism to perform global roll-up analysis in an environment where data is widely distributed across multiple locations without using DistCp to move the data to a central site. This would be in violation of these regulations and can only be addressed with extensive programming, which is costly and undermines flexibility. Timely analysis of data from multiple locations is problematic. With Hadoop s standard architecture data has to be copied from dozens of sites to a central location for analysis in a difficult, time consuming and unreliable process. As more customers (and more cybercriminals) operate in multiple countries, fast, reliable data analysis close to the source, with reliable global consolidation is a critical requirement. //02

4 Solution After evaluating products that support Hadoop, the bank decided to deploy Non-Stop Hadoop to replicate data across data centers and improve the regulatory compliance, performance, scalability, and flexibility of its Hadoop applications. The bank found that Non-Stop Hadoop offered significant benefits in the areas outlined below. Improved Regulatory Compliance with Continuous Availability and Guaranteed Data Consistency During the implementation period, the bank s IT staff discovered additional advantages of Non-Stop Hadoop, including continuous availability and guaranteed data consistency. Numerous banking regulations mandate that specific data must be available at all times. This requires bulletproof backup and disaster recovery capabilities. It also means that backup copies of data must remain consistent with primary copies. Non-Stop Hadoop features an active/active architecture that performs multi-directional data replication automatically, continuously, and reliably across multiple sites. Data is always stored in more than one location, and is current and consistent across data centers. This means: Stringent disaster recovery objectives are met. If a regional data center fails, data is available at another location with virtually zero downtime and no data loss. Data consistency requirements are met, because users and applications at different locations always have local network-speed read/write access to identical data sets, subject to any selective replication policies in place. Administrators no longer spend time monitoring data movement or trying to reconcile backup and primary clusters. Cost Savings Through 100% Utilization of Compute Resources Non-Stop Hadoop s active/active architecture guarantees 100% use of compute resources. All servers at all locations are fully readable and writable. Because of this, the bank has been able to eliminate spending on read-only backup servers for failover and disaster recovery. Improved Compliance with Data Protection Regulations Non-Stop Hadoop offers selective replication capabilities that enable the bank to remain compliant with data privacy and data protection regulations in the various countries it operates in, while still enabling the bank to perform global roll-up analysis. Because Non-Stop Hadoop is data center-aware, data does not have to be moved to a central location for analysis. Applications can run from any location and access the data regardless of how it is distributed across the organization. This feature is particularly valuable for applications such as fraud detection, marketing analysis, and risk modeling, which need to detect patterns in vast global data sets. //03

5 Maximizing Performance and Lowering Costs with Cluster Zoning Non-Stop Hadoop adds a cluster zoning capability to Hadoop. The bank now creates virtual clusters in key data centers. These virtual clusters within a cluster contain the high-end servers with the most RAM and processing power. The bank runs real-time in-memory analytics applications for fraud detection and anti-money laundering in these virtual clusters, and less time-sensitive processes on commodity servers within the same Hadoop cluster. This approach helped the bank achieve its goal of increasing the performance of its critical real-time analytics applications, without a massive increase in hardware spend, while still meeting quality of service requirements for all applications. Without cluster zoning, the bank would have to invest in high-end servers throughout the cluster. Fast, Reliable Data Replication and Access with Multi-Data Center Ingest With Non-Stop Hadoop, data can be ingested at multiple locations simultaneously, and then automatically replicated to any number of locations, without the administrative overhead and risk of error involved in using DistCp to transfer data to a central location for analysis. Client applications have LAN-speed read/write access to the same data everywhere. Fraud analysts in Beijing, Brisbane, and Buenos Aires can have access to the same data at the same speed as their counterparts in New York, Tokyo, and Frankfurt. Because the bank s customers (and attackers) are multinational and highly distributed, this fast, reliable access to local as well as global data is critical for applications such as new account approvals, fraud detection, customer service, risk modeling, liquidity management, and regulatory compliance. Results Non-Stop Hadoop has produced concrete benefits for the bank, including: Continuous availability and guaranteed data consistency make it easier to comply with stringent regulations for bulletproof backup, automated recovery and data consistency without administrator overhead. Multi-data center ingest has enabled real-time decision making in remote offices far from the bank s large data centers. 100% utilization of compute resources eliminated the expense of underutilized servers for backup and disaster recovery. Cluster zoning enables quality of service to be maintained for all applications by isolating the most resource intensive applications to the servers best suited to support them, without requiring high-end servers throughout the cluster. This feature, in combination with 100% use of compute resources, reduced hardware costs by over 35%. Selective replication allows applications to utilize global data without moving customer information across borders, in violation of data security and privacy protection regulations. Straightforward implementation, requiring no changes to applications, networking or other infrastructure. //04

6 In addition, Non-Stop Hadoop has overcome the concerns of IT executives about deploying mission-critical applications with Hadoop. This has opened the way to tens of millions of dollars in new revenue, dramatically reduced losses due to fraud, and enabled significant savings on infrastructure spending in excess of 35%, compared to traditional Hadoop solutions. Today the bank is investigating how Hadoop and Non-Stop Hadoop can be applied to other business applications that deal with the flood of real-time data that will be generated by mobile payment apps, and how to improve decision-making with unstructured data from mobile apps and social media. For more information, read Five Reasons Your Hadoop Deployment Needs WANdisco Or visit wandisco.com Copyright WANdisco, Inc. All rights reserved. Hadoop is a trademark of the Apache Software Foundation. All other trademarks and copyrights are the property of their rightful owners. //05