What is a Data Lake, anyway? Alec Gardner, GM Advanced Analytics, Teradata ANZ Wednesday 10 th June 2015

Transcription

1 What is a Data Lake, anyway? Alec Gardner, GM Advanced Analytics, Teradata ANZ Wednesday 10 th June 2015

2 A large objectbased repository that holds data in its native format store all data present and future and create a centralised data archive location. One of the Big Data labels that we risk over-loading to complete abstraction is the idea of a "Data Lake Sometimes called the bit bucket or the landing zone As more and more applications are created that derive value from new types of data the Data Lake forms All Water and Little Substance Teradata

3 One of the Big Data labels that we risk over-loading to complete abstraction is the idea of a "Data Lake Teradata

4 Big Idea #1: store all data (whatever all means) Big Idea #2: un-washed, raw data (NoETL / late-binding) and at least part of the discussion sounds familiar Big Idea #3: resolve the nagging problem of accessibility and data integration Data accessibility and integration? Isn t that what the Data Warehouse is for? Teradata

5 Is the Data Lake a new architectural construct? Or are we building next-generation data silos? Simple, single subject area Dimensional Data Marts with all of the dimensions pre-joined to the fact table? One-perworkload / application? Is this really the future of Enterprise Analytics? Or circa 1995 silo, departmental Decision Support Systems warmed-over and re-platformed? Teradata

6 Teradata Up-front data modelling and data integration are difficult, time-consuming and expensive; maybe we just shouldn t bother with them any more?

7 but there are no free lunches in Information Management merely more and different options Explicit, or implicit, there is always, always, always (at least one) schema; pay me now, or pay me later (and over and over) Teradata

8 Gartner Logical Data Warehouse Forrester Enterprise Data Hub Teradata Unified Data Architecture Big Data Are Plural For the foreseeable future, we will need multiple Information Management strategies - and multiple Information Management technologies Integration becomes a critical concern Teradata

9 The new Big Data The big 5 challenges of making the transactions, to interactions journey #1: The requirement to manage multistructured data and data whose structure changes continuously means that there is no single Information Management strategy that works equally well across the entire Big Data space. #2: Understanding Interactions requires path / graph / time-series Analytics in addition to traditional set-based Analytics, so that there isn t a single parallel processing framework or technology that works equally well across the entire Big Data space. #3: The economic challenge of capturing, storing, managing and exploiting Big Data sets that may be large; getting larger quickly; noisy; of (as yet) unproven value; and infrequently accessed. #4: There might be a needle in one of these haystacks - but if it takes 6-12 months and $1M just to go look, I ll never know. #5: Getting past so what to drive real business value (because old business process + expensive new technology = expensive, old business process) 10

10 The Logical Data Warehouse is the industry s adaptation to Big Data How will you deploy? How many / which platforms will you need? How will you integrate them? And which data need to be centralised and integrated? The Enterprise Data Warehouse Era The Logical Data Warehouse (a.k.a.: Unified Data Architecture) Era 1 Multi-structured data 2 Interaction / observation Analytics Flat / falling IT budgets, exploding data volumes Agile Exploration & Discovery Give me integrated, high quality data. 5 Operationalisation Centralise and integrate the data that are widely reused and shared, but integrate all of the analytics. 11

11 Teradata

12 The Data Lake as it should be: a centralised, consolidated store of raw data from multiple sources Agile acquisition of raw, multi-structured data efficient non-relational computation and cost-effective storage of large and noisy data-sets Now that is new, interesting and potentially very, very useful Teradata

13 Repeatable process and reproducible results are the basis of experimental science Data. Science. successful, real-world laboratories are not ungoverned environments Teradata

14 Any chain is only as strong as its weakest link The Enterprise Information Value Chain Where did this data come from and when was it created? What did its creator believe that it represented? Full copy or delta file? Part of a collection? Etc., etc., etc Teradata

15 STOP PRESS: Laws of Physics Unchanged by Big Data! Left to its own devices, does nature tend to give us a single, beautiful lake? Or a messy patchwork of lakes, plural? The new information management strategies and technologies are not a cure for information entropy; a well-ordered Data Lake will not just spontaneously emerge from a collection of consolidated Hadoop applications Teradata

16 Architecture and Technology are only enablers Web / clickstream Who navigates to the website, what do they do in each session and then afterwards within other channels? Sentiment What are customers saying about the company / products / services on social media sites? Voice/text Who is complaining to the call center & about what? / Graph Which brokers are colluding to rig markets and with whom? Process / Path Analytics What s the optimal process for claims or collections activity? Teradata

17 Teradata All Data And The Big Bit Bucket Myth

18 Operationalising Exploration & Discovery Insight Type 1 Discovery Analytics identifies a broken process; Fix the business process and you are done; E.g.: Large US Telco used Path Analytics to understand pathto-call-centre journey. Teradata Unified Data Architecture Type 2 Discovery Analytics enables us to identify / extend an existing model that is already being scored in the IDW; E.g.: a leading US Retail Bank was able to enhance its existing churn models by understanding the path to churn. Type 3 New model is dependant on data that is not yet available in the IDW and/or on Analytics (e.g.: path, graph, time-series, etc.) that cannot easily be supported there; Operationalisation requires run-time integration of multiple, distributed components. QueryGrid Integration Technology Teradata

19 Teradata Summary and Conclusions

20 Teradata