HADOOP AND MAINFRAMES CRAZY OR CRAZY LIKE A FOX? Mike Combs, VP of Marketing mcombs@veristorm.com

Transcription

1 HADOOP AND MAINFRAMES CRAZY OR CRAZY LIKE A FOX? Mike Combs, VP of Marketing mcombs@veristorm.com

2 The Big Picture for Big Data 2 The Lack of Information Problem The Surplus of Data Problem The 3 V s of Big Data Volume: More devices, higher resolution, more frequent collection, store everything Variety: Incompatible Data Formats Velocity: Analytics fast enough to be interactive and Useful (Doug Laney, Gartner Research)

3 Big Data: Volume 3 SDDS telescope, 80 TB in 7 years LSST telescope, 80 TB in 2 days

4 Big Market, Big Growth 4

5 Big Data: Variety 5 20% is Structured 80% is Unstructured Tabular Databases like credit card transactions and Excel spreadsheets Web forms Pictures: Photos, X-rays, ultrasound scans Sound: Music (genre etc.), speech Videos: computer vision, cell growing cultures, storm movement Text: s, doctor s notes Microsoft Office: Word, PowerPoint, PDF

6 Big Data: Velocity 6 To be relevant, data analytics must be acted upon in a timely fashion Results can lead to other questions, and so the solutions should be interactive

7 Big Data Industry Value 7

8 Increasing needs for Detailed Analytics 8 Baselining & Experimenting Parkland Hospital analyzed records to find and extend best practices Data Sharing US Gov Fraud Prevention shared data across departments Segmentation Dannon uses predictive analytics to adapt to changing tastes in yogurt Decision-making Lake George ecosystem project uses sensor data to protect $1B in tourism New Business Models Social media, location-based services, mobile apps

9 Hadoop Family & Ecosystem 9 Hadoop solves the problem of moving big data Eliminates interface traffic jams of getting data from a large disk Eliminates network traffic jams of getting data from many disks Hadoop divides and moves the work instead Hadoop divides the job across many servers and sends the work to them Apache Hadoop is an opensource platform Hadoop includes HDFS File-based, unstructured, massive MapReduce Distributes processing and aggregates results (queries or data loading) Pig Programming language Hive Structure with SQL-like queries Hbase Big table, with limits Flume Import streaming logs Sqoop Import from RDBMS

10 Typical Hadoop Cluster 10 NameNode Master, directs slave DataNodes, tracks file block storage, overall health Secondary NameNode Backup JobTracker Assigns nodes to jobs, handles task failures Slave Nodes DataNode IO and backup TaskTracker Manages tasks on slave; talks with JobTracker

11 11 Today s Big Data Initiatives: Transactions, Logs, Machine Data Unstructured data

12 Transaction Data = Mainframe Computers 12 Mainframes run the global core operations of 92 of top 100 banks 21 of top 25 insurance 23 of top 25 retailers Process 60% of all transactions Mainframe computers are the place for essential enterprise data Highly reliable Highly secure IBM s Academic Initiative 1000 higher education institutions In 67 nations Impacting 59,000 students However, mainframe data uses proprietary databases which must be translated to talk to current formats

13 The ROI Behind the Hype 13 The most relevant insights come from enriching your primary enterprise data Sensitive, Structured Customers, Transactions,... Semi-public, Unstructured Call center data, Claim Systems, Social,

14 14 Bring Analytics to the Data rather than the Data to the Analytics Extract, Transform and Load (ETL) 1TB ETL per day, Initial copy plus three derivatives costs > $8 million over 4 years Operational applications Data transfer Analytical applications Multiple copies of data Transaction and analytics isolation Significant compute power Source: CPO internal study. Assume dist. send and load is same cost as receive and load.. Also, assume 2 switches and 2 T3 WAN connections. Before we even start with a workload evaluation, we need the answer to one important question: Where s the data located? The ETL Problem, Clabby Analytics

15 Obstacles to Include Mainframe Data 1/ Data Governance as the data moves off z/os operational systems 2/ Data Ingestion from z/os into Hadoop (on or off platform) is a bottleneck (MIPS & ETL cost, Security around data access and transfer, Process Agility) Lead to key requirements: Existing security policies must be applied to data access and transfer. There needs to be high speed / optimized connectors between traditional z/os LPARs and the Hadoop clusters Ability to serve data transparently into Hadoop clusters on mainframe AND on distributed platform

16 The Dilemma: Ease of Access vs. Governance DB2 VSAM Extract from proprietary formats. Transform IMS z/os Logs Aggregate or summarize. Staging Load JCL, DB2, HFS, VSAM, IMS, OMVS, COBOL Copybooks, EBCDIC, Packed Decimal, Byte ordering, IPCS, z/vm, Linux on z Hadoop, MongoDB, Cassandra, Cloud, Big Data Ecosystem, Java, Python, C++, Interface skills

17 The Dilemma: Ease of Access vs. Governance DB2 VSAM Extract from proprietary formats. Transform IMS z/os Logs Aggregate or summarize. Staging Load Push from IT OR Pull from Business

18 vstorm Enterprise Mainframe data to Mainstream 18 z/os DB2 VSAM IMS Linux vstorm Connect zvm zdoop Logs IFL IFL System z

19 vstorm Enterprise (vstorm Connect + zdoop) 19 z/os DB2 VSAM IMS Logs Linux vstorm Connect zvm IFL IFL zdoop MapReduce HDFS, Hive A secure pipe for data Data never leaves the box RACF integration no need for special credentials Data streamed over secure channel using hardware crypto Easy to use ingestion engine Native data collectors accessed via graphical interface Light-weight; no programming required Wide variety of data sources supported Conversions handled automatically Streaming technology does not load z/os engines or require DASD for staging System z Templates for agile deployment Spin up new nodes on demand An ideal cloud deployment platform Mainframe efficiencies

20 Select HDFS or Hive destination for copy Select source content on z/ OS Browse or graph content 20

21 Velocity, the 3 rd V z/os Linux DB2 Data Metadata Metadata Data Hive 2 Billion records transferred, converted and analyzed in 2 hours over 2 IFLs Time IFLs 6 weeks IBM benchmark validated linear scalability across the standard Hadoop stress tests

22 Security: vstorm Enterprise is integrated with RACF System z RACF LDAP server z/os CP(s) LDAP Client z/vm / Linux IFL(s) IFL(s) vstorm Enterprise Servlet Applet HDFS, HIVE vhub IFL(s) Integration with RACF to extend z/os security domain onto zdoop Leverage HiperSocket for secure data transfer (vs. data traveling over the network) Data in flight encryption via SSL Data at rest encryption via Hardware Crypto

23 Address the Skill Gap for new applications 23 Enable the developer community to take advantage of the enterprise primary data in a model they understand Bring your System administrators and developers together Provide easy & controlled access to mainframe data Familiar environment: Linux & Hot technology: Hadoop Procedural programming: Java, Python, JavaScript, PIG

24 Get Started Install the vstorm Enterprise in 2 hours Rapid end-to-end processing of enriched customer and transaction data Detecting fraud Enhancing insurance policy enforcement Reducing healthcare costs Improving claims response time 24

25 25 Thank you!??? Questions?

26 Financial Services Use Case 26 Problem Solution Benefits High cost of searchable archive on mainframe $350K+ storage costs (for 40TB) MIPS charges for operation $1.6M+ development costs due to many file types, including VSAM man-days effort and project delay Move data to Hadoop for analytics and archive Shift from z/os to Linux to reduce MIPS Use IBM SSD storage Use IBM private cloud Softlayer Tap talent pool of Hadoop ecosystem Reduction in storage costs Dev costs almost eliminated Quick benefits and ROI New analytics options for unstructured data Retains data on System z for security and reliability

27 Health Care Use Case 27 Problem Solution Benefits Relapses in cardiac patients One size fits all treatment $1M+ Medicare readmission penalties Sensitive patient data on System z VSAM No efficient way to offload Identify risk factors by analyzing patient data* Factors used to predict likely outcomes 31% reduction in readmissions Estimated $1.2M savings in penalties No manual intervention No increase in staffing 1100% ROI on $100K * System z VSAM database requires special skills to access without vstorm

28 Public Sector Use Case 28 Problem Solution Benefits Mismanaged assets led to shabby neighborhoods, publicity problems, often law & order issues e.g. broken lights, abandoned bikes, tsunami sirens Post-2008 austerity measures low budget Asset data System z IMS* based no efficient offload Crowd source problem reporting cell phone photo, social media, GPS data Integrate social media problem reports with asset / governance data on System z on Hadoop for System z (conforming to regulations) Software costs of $0.4M compares to $2M consulting engagement proposal Better maintained neighborhoods projected to yield $5.6M higher taxes in first year * System z IMS database requires special skills to access without vstorm

29 Retail Use Case 29 Problem Solution Benefits Streams of user data not correlated e.g. store purchases, website usage pattern, card usage, historical customer data Historical customer data System z VSAM & DB2 based no efficient, secure offload HDFS securely populated with historical customer data, card usage, store purchases, website logs Splunk scores customers based on the various data streams* High scoring customers offered coupons, special deals on website 19% increase in online sales in the middle of retail slowdown Over 50% conversion rate of website browsing customers (shopping cart to sale) Elimination of data silos since now analytics cover all data no more reliance on multiple reports / formats