Big Data & The Cloud

Transcription

1 DAMA NY CHAPTER PRESENTATION Big Data & The Cloud Extreme Performance Data Warehousing Inside Of The Cloud Robert J. Abate, CBIP, CDMP Solutions Principal, EIM & Analytics Practice EMC CConsulting sut January 19 th,

2 DAMA NY CHAPTER PRESENTATION Big Data & The Cloud AGENDA Background & Definitions The Challenge Architectural t Solutions To Big Data It s A Brave New World Example Case Studies Open Discussion 2

3 Background & Definitions 3

4 Big data will represent a hugely disruptive force during the next five years enabling levels of insight that are currently unachievable through any other means Gartner May

5 We Are Awash In Data In the information age, every organization is in the data business Data is growing exponentially, so are the challenges Complexity is causing insight to be lost Source: IDC Digital Universe White Paper, Sponsored by EMC, May

6 Pictorial Representation Of Information 6

7 Big Data: More Than Just About Volume Consider: Master Data, Fidelity, Complexity, Validity, Perishability, Linking Data Structured Transactional Data: POS transactions, call detail records, credit card transactions, shipping updates, purchase orders, payments, shipments, account transactions Unstructured Data: Web logs, newsfeeds, social media, geolocation, mobile, consumer comments, claims, doctor s notes, clinical studies, images, video, audio Device-generated Data: RFID sensors, smart meters, smart grids, GPS spatial, micro-payments Social Sensor/ locationbased Web traffic Variety Industryspecific Velocity Documents Transactional Data Video Volume Text Complexity Audio Images Smart Grid 7

8 The Typical BI/DW Environment Today 8

9 Big Data s Potential For Actionable Insight Today s Situation Less than 10% of the enterprise s data Rear-view mirror reporting, dashboards d and analysis Weeks, months, or even quarters old Incomplete, inaccurate, and disjointed data Architectures and methods that take 6 to 18 months to exploit Big Data Ramifications Vast majority of available sources and external data Forward looking or Windshield-view i predictions with recommendations Real-time near real-time Correlated, high confidence, governed data Vastly accelerated time to market 9

10 Time Really is Money! Value The Time Value Curve THE TIME VALUE CURVE Dr. Richard Hackathorn, Bolder Technology, Inc., All Rights Reserved. Used with Permission. Business Event Value Lo ost Capture Latency Data Ready For Analysis Analysis Latency Information Delivered Decision Latency Action Time Action Taken Time Data Lifecycle 10

11 Data Is Coming At Us Faster In a recent TDWI survey of 450 CIO s 17% have a real time data warehouse 90% plan on having a real time warehouse 75% will replace to get to a real-time solution REAL TIME IS A RAPIDLY BECOMING A NECESSARY FOUNDATION TO A DATA SOLUTION AND WITHOUT ARCHITECTURE THERE IS CHAOS! 11

12 Data Is Coming From All Directions Data is now commonly entering into the enterprise from external sources Government (Census, Revenues, ) Neilson, NPD Group (Sales) Bloomberg, NYSE (Financial Position) Experian, TransUnion, Equifax (Credit Reporting) Google Maps, MapInfo (Geospatial, ) Radian 6, Biz360, (Client Trend Data) Etc. 12

13 Need For Data Trust Compliance with laws Revenue Canada, Sarbanes Oxley [SOX], BASIL II, HIPAA, etc. Lack of confidence in the data Reports utilizing same data do not report same totals or computations Data not defined d and readily available Multiple sources of data have to be rationalized at each project start-up thereby wasting valuable time & $ on every yproject Data timeliness Manual process to collect, analyze and provide results Data integrityit Unknown filters, varying calculation/computations, fields used for data not indicative of field names, data passed along from one person to another to another to another.. 13

14 Summation Of Challenges We Are Observing Business mandate to obtain more value out of the data (get answers) Variety of sources, amounts, types and granularity of data that customers want to integrate is growing exponentially Need to shrink the latency between the business event and the data availability for analysis and decision-making Advancing agility of information is key Need for Data trust and Compliance with regulations 14

15 The Challenge Of Big Data 15

16 Old Journey To Information Maturity [EIM] Data Chaos Same type of data means different things in different systems Ex: AT&T is the same as AT&T Inc PROCESSES Master Data Publish and Subscribe to master data Ex: Single view of customer across all information Data Discovery Data systems Governance Data Integration Data Mining Data Analytics Analyzing the data. Looking for trends and correlations Data Chaos Defined Data Master Data Integrated Information Data Analytics Business Optimization TOOLS Data Discover Metadata ETL Suite BI / DW / OLAP Defined Data Integrated Predictive Define common meanings. Ex: Determine the sources, types, and properties of grouped (i.e.: customer) records Information Bring metadata together with information for reporting (BI) and warehousing (drilling and hierarchies). Information Using the analyzed data to optimize operations Wiki Type Sharing Of Self- Provisioned Environments Atomic Data Analytics 16

17 The Information Issue Is Too many organizations are not using information to its full advantage: 1 in 3 business leaders frequently make critical decisions without the information they need 1 in 2 business leaders do not have access to the information across their organization needed to do their jobs. 3 in 4 business leaders say more predictive information would drive better decisions Source: IBM Institute for Business Value, March

18 Information Trust & Business Alignment Harris Interactive recently polled 23,000 U.S. employees and found Only 37% said they have a clear understanding of what their organization is trying to achieve and why Only one in five was enthusiastic ti about their team and the organization s / corporation s goals Only one in five said they have a clear line of sight between their tasks and their team and organization s goals Only 15% felt that their organization fully enables them to execute key goals Only 20% fully trusted the organization they work for 18

19 Viewed Using An Seasonal Analogy If a football team had these players on the field: Only 4 of the 11 players on the field would know which goal is theirs Only 6 of the 11 would care Only 3 of the 11 would know what position they play and what they are supposed to do 9 players out of 11 would, in some way, be competing against their own team rather than the opponent 19

20 Perceived Complicated Landscape BI/DW is perceived as not enabling the business Inhibitor to corporate progress IT systems cannot be changed fast enough to meet market demands, seize opportunity or comply with a new requirement. Weak alignment between IT and business strategy Marked by an intractable language barrier. Business not always sure what Information or Dimensions they want or need How can IT provide without requirements? BI/DW is not known as the source of innovations The complexity of systems has caused BI/DW to be reactive rather than proactive Silo d solutions, db s and applications with trapped business rules Multiple sources of information and no single truth No Architectural Blueprints to the enterprise 20

21 The Business Intelligence Maturity Model 21

22 Advancing The Maturity Of Information 22

23 The big data impacts to both business and IT are significant; early adopters will fundamentally change their industries Business Expectations IT Ramifications More agile, more real-time, more accurate decision-making Predict and spot changes in dynamic and volatile markets Deeper understanding of customer preferences and behavior Greater fidelity in risk assessment and compliance enforcement Enhanced user experience that delivers insights to any device Operationalization of data scientists and analytic insights Tools and processes for data quality, governance, and security Cloud for self-service, collaboration, agility, and cost reduction Big data poses a major opportunity for CIOs to drive added value for the business, by deriving insights and identifying patterns from the huge amounts of data available Through 2015, organizations integrating high value, diverse new information sources and types into a coherent information management infrastructure will outperform industry peers financially by more than 20% Source: Gartner "The New Value Integrator," Insights from the Global Chief Financial Officer Study July

24 Architectural Solutions For Big Data 24

25 Big Data Requires Change Consider 100 GB would store the entire US Census DB basic information set for every living human being on the planet: Age, Sex, Income, Ethnicity, Language, Religion, Housing Status, Location into a 128 bit set That equates to about 6.75 millions rows of about 10 collumns Consider the Large Hadron Collinder at CERN Expected to produce 150,000 times as much raw data each year 25

26 The Big Change In Technologies Consider that Relational technologies were invented to get data in and organized, not designed nor organized to get it out RDBMS s were designed for efficient transactions processing on large data sets Adding, Updating Searching for & retrieving small amounts of data [2] Source: ACM Website The Pathologies of Big Data, Adam Jacobs, 7/6/09 26

27 Data Warehouses Were An Answer DW was classically ll designed d as copy of transaction data specifically structured for query and analysis General approach is bulk ETL into a DB designed for queries Big data changes the answer Traditional RDBMS-based dimensional modeling and cube-based based OLAP turns out to be to slow or to limited to support asking the really interesting questions of warehoused data [2] To achieve acceptable performance for highly order-dependent queries on truly large data, one must be willing to consider abandoning the purely relational database model [2] [2] Source: ACM Website The Pathologies of Big Data, Adam Jacobs, 7/6/09 27

28 Voluminous Data Sets What makes large data sets are repeated observations over time/space Web log has M s visits over handful pages Retailer has 10K products, M custs, but B trans Hi-Res Scientific like fmri 1K GB per view Large datasets t Spatial or Temporal dim s Cardinalities (distinct observations) is usually small with regard to total # of observations 28

29 Technology Solutions Appeared 29

30 Lets Talk Technical Solutions Sequential and/or Distributed File-Based Solutions Oracle Exadata, Hadoop, etc. Columnar (compression) )/ Multi-Level Tables Solves challenge of retrieving entire row Par-Excel, Vertica, Sybase, etc. Distributed MPP Teradata, Greenplum, etc. Polymorphic Combination of Columnar & MPP 30

31 Finding Answers Sequentially With OLTP Random access is slower than sequential The advantage gained by doing all data access in sequential order is often 4x 10x Many orders of magnitude! [2] Source: ACM Website The Pathologies of Big Data, Adam Jacobs, 7/6/09 31

32 Distributed File: Partitioning With OLTP Partitioning can solve challenges of data growth, but true distributed processing utilizing MPP is best (author s opinion) 32

33 Distributed File: Partitioning Viewed Q: What was the total transactions (sales) amount for May 20 and May ? Sales Table 5/17 Select sum(sales_amount) From SALES Where sales_date between to_date( 05/20/2009, MM/DD/YYYY ) And to_date( 05/22/2009, MM/DD/YYYY ); Only the 2 relevant partitions are read 5/18 5/19 5/20 5/21 5/22 Source: Extreme Performance With Oracle Data Warehousing 33

34 Distributed File: Open Source (Hadoop) Apache Hadoop is a software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data Hadoop was inspired by Google's MapReduce and Google File System (GFS) papers. Hadoop is a top-level Apache project being built and used by a global community of contributors using the Java programming language. Yahoo! has been the largest contributor to the project, and uses Hadoop extensively across its businesses. Source: Wikipedia Hadoop 34

35 Distributed File: Hash-Based Distribution In a hash-based data distribution, the data is distributed across multiple platforms for parallelism li of queries 35

36 Columnar: Storage In a table with say 256 columns, a lookup will retrieve all the data in the row (disk bound) Columnar storage reduces this I/O bandwidth by storing column data using compression State (50 combinations stored) Master (compressed) table has pointers to State Source: Vertica Website 36

37 Columnar: Multi-Level Table Partitioning In multi-level table partitioning, data distribution occurs across multiple platforms in segmented tables for distribution of columnar queries This reduces the amount of work performed by each platform 37

38 MPP Shared Nothing Architectures Extreme scalability Elastic Expansion & Self-Healing Fault-Tolerance Unified Analytics Source: Greenplum Database 4.0: Critical Mass Innovation, White Paper, August

39 MPP Shared Nothing Architectures Source: Greenplum Database 4.0: Critical Mass Innovation, White Paper, August

40 The Ideal MPP Shared Nothing Poly-Morphic Storage Tabular, Columnar, NoSQL, etc. 40

41 It s A Brave New World 41

42 From the Old Stack to a New Ecosystem: Drivers for Change Many new data sources (organic growth, data services, M&A) Impractical to add new data sources because of tightly coupled pipeline More unstructured t data, including social media Lack of access to unstructured data; need analytics and classifiers that operate on it Less up front data integration Can t assume data is pre-integrated have to be able to locate and to query federated sources of data and content More need to track and leverage metadata Metadata is fragmented, jailed and inconsistent need agile, community approach Need for flexible, agile data structures Current structures are too rigid, and too close to the sources or the business reports More emphasis son dynamic views for purpose pose Need dynamic planning, creation and structuring of views that support analytics Information governance and management in a federated, regulated world Need flexible policy expression and enforcement, not just at point of access 42

43 An Information Platform with New DNA To Promote Agility, Business Value and Community 1. Coordinated ingestion of diverse information, changes, events 2. Metadata driven processing and management 3. Nuanced optimization on demand, multi-source, matching information needs 4. Broader reach of query contextual search, federation, materialization 5. Freedom from imposed information structure roll your own structure! 6. Navigation through information contextual, faceted, multi-dimensional 7. Visualization of information heat, clouds, clusters, flows 8. New data paths engendered by patterned consumption of entities 9. Reasoning about data set location, derivation, freshness, and obligations 10. User empowerment collaboration and talent development 43

44 Businesses Want Integrated, Timely Information for Purpose Area Latency Enrichment Query Federation Source Revolution Microbatch is the new Batch Tagging is the new Transformation Query is the new ETL Query Director is the new Query Optimizer Purposeful View is the new Master 44

45 Some Of The Newer Trends In Big Data Powerful Analytics What if, What will happen next, Self-service analytics? Build your own sandbox of data a Data Cloud Surrounded Warehouse Data Virtualization Abstracting the data from the systems, it complements existing data warehouses Many times the size of structured warehouse Provides for rapid analytic iterations 45

46 When You Link Structured & Unstructured Information You Get 46

47 Powerful Analytical Engines What is the best price to sell my product? 47

48 How Do I Do This? 48

49 How Do I Do This #2? 49

50 How Do I Do This #3? 50

51 Visualize The Information 51

52 Analytics: A Picture Is Worth A 1,000 Words 52

53 Data Virtualization Example 53

54 Data Virtualization In Practice 54

55 Enterprise Big Data Cloud 55

56 The Future Of Data Warehousing? The Ideal Abate Enterprise Data Cloud Truly Virtualized Data Environment Extreme Scale, Elastic Expansion Automated Metadata Discovery, Classification & Tagging Linearly Scalable Add 1x and get 2x performance Self Service Provisioning Single Point Of Management Resource utilization optimization Secure, Unified Data Access Single Point of Entry Portal based sharing of data sandboxes (wiki-type) Reduce TCO By Eliminating Excessive Licensing Fees Use of open source community to improve solution 56

57 Example Case Studies 57

58 BIG DATA ANALYTICS USE CASE Telecomm eco Provider Learns A Lesson Before investing $M of dollars on infrastructure, a provider learned where to invest their monies that would payoff Challenge 100TB Traditional EDW, Single Source Of Truth Operational Reporting & Financial Consolidation Heavy Governance And Control Unable To Support Critical Business Initiatives Customer Loyalty And Churn The #1 Business Initiative From The CEO Enterprise Data Cloud Architecture-Based Solution Extracted Data From EDW & Other Sources Generated Social Graph From Call Detail And Subscriber Data Within 2 Weeks Found Connected Subscribers 7X More Likely To Churn Than Average Users Now Deploying 1PB Production 58

59 BIG DATA ANALYTICS USE CASE Drive Multi-channel Campaign Optimization Retailer increases in-flight multi-channel effectiveness with customer and product insights HIGH Likeliho ood Of Conversi ion Legacy System Advanced Analytics Monitor crosschannel product sales effectiveness Big Data Analytics Integrate t customer behavioral data with social media sentiment data to yield new market, product and campaign insights LOW 59

60 BIG DATA ANALYTICS USE CASE Innovate With Big Data Analytics Big Data Analytics Accelerate Health Care 2.0 for Evidence-based Care Provider HIGH Legacy System BI Reporting Advanced Analytics Quality of Care Delivering 10 Years Of Data In Seconds Associative Rule Mining and User Clustering Improves Pathways Big Data Analytics External Data Sources Enable Personalized Medicine LOW Treatment Pathways ays on Summary Data Treatment Pathways ays on All the Data TRADITIONAL DATA LEVERAGED BIG DATA LEVERAGED 60

61 Open Exchange Of Ideas Speaker Contact Information: Robert J. Abate, CBIP, CDMP (201)

62 Credits To Quoted Authors Adam Jacobs is senior software engineer at 1010data Inc., where, among other roles, he leads the continuing development of Tenbase, the company s ultra-high-performance analytical database engine. He has more than 10 years of experience with distributed processing of big datasets, starting in his earlier career as a computational neuroscientist at Weill Medical College of Cornell University (where he holds the position of Visiting Fellow) and at UCLA. He holds a Ph.D. in neuroscience from UC Berkeley and a B.A. in linguistics from Columbia University. (QUOTED FROM: The Pathologies of Big Data, 7/6/09) Bill Schmarzo has over two decades of experience in data warehousing, BI and analytic applications (Metaphor Computers, 1984). Bill authored the Business Benefits Analysis methodology that links an organization s strategic business initiatives with their supporting data and analytic requirements, and coauthored with Ralph Kimball a series of articles on analytic applications. Bill has served on The Data Warehouse Institute faculty as the head of the analytic applications i curriculum. Bill was VP of Analytics at Yahoo where he was responsible for the development of Yahoo s Advertiser and Web Site analytics products, including the delivery of actionable insights through a holistic user experience. For Business Objects, Bill oversaw the Analytic Applications business unit including the development, marketing and sales of Business Objects industry-leading analytic applications. Donald Sutton has over 20 years experience in Data Architecture, Analysis, Modeling, ETL, Implementation and Integration in the areas of Data Entry (OLTP) or ERP and 3rd Party COTS Applications, Operational Data Store (ODS), Master Data Store (MDS), Data Warehouse (DW) and Data Marts (DM) while providing Business Intelligence (BI) from multiple sources above. Passionate and motivated about sound design of data structures in all different data layers and the representation and transformation ti of data with the accounting and governance of data throughout h t all data layers while Providing Business Intelligence (BI) and analytics with Key Performance Indicators (KPI) along with business modeling in translating business requirements to data requirements. (QUOTED FROM: Current Warehousing Environment & Analytics Visualizations) 62

63