Big Data Case & Roadmap - Requirements

Transcription

1 Building Big Data Business Case & Roadmap DAMA Wisconsin November 13 th 2014

2 2

3 The Buzz 3

4 DefiniKon Data in a corpora+on, internal or external that cannot be processed using tradi+onal data management techniques and technologies can be classified as Big Data. 4

5 What s so Big about Big Data Velocity Volume Variety Viscosity Veracity Complexity Ambiguity 5

6 Data Collected Today Beyond OLTP 12+ TBs of tweet data every day? TBs of data every day 25+ TBs of log data every day 6

7 Future of Data 7

8 The QuesKon s Is there real value? Is there real insights? How much of the data is garbage? What will I know new? This is too large to comprehend Who can validate the source of the data? What is the lifecycle for this data? What analykcs will this help drive? 8

9 Tapping into the data! Structured data used today Narrow structures, defined API, transac6onal data,metrics and applied measures Big Data existing across the enterprise that can be made available to business Big structures, wide and deep, no defined business metadata, no applied metrics, no defined context 9

10 Big Data in CorporaKons Internal Documents Call Center Data Project Data Audio, Video 8k,10k & Financial Data Contracts Sales & MarkeKng Data External Forums Twiaer Facebook Instagram Magazines & Media Public Domain Data 10

11 The Imbalance Data Warehouse Enterprise Data Internet CRM SCM, ERP, HRM IT Provisioned EDW Content Shadow IT Provisioned AnalyKcs Call Center (Voice) 11

12 A Balanced Pordolio M D M & M E T A D A T A Visualiza6on Analy6cs Seman6cs (Taxonomies, Ontologies, Metadata Libraries) Transforma6on Raw Data (Big, Small, Fast, Slow, Unstructured, Semi- Structured) G O V E R N A N C E INFRATSRUCTURE PLATFORMS 12

13 User Trends ImpacKng Business Internet Mobile Social Media CrowdSourcing Peer and Expert RecommendaKons CompeKKve Threats 13

14 SenKments - Example I personally love the texture of this and while the price is much higher than I would like to pay. Typically I would pick up at walgreens in the morning. I usually pick up a jamba Juice in the morning but this is a liale cheaper and packs more protein. The flavor is bearable, even tolerable, but it's the griay aler taste that ruins the over all experience. I don't think that I would purchase it again... Why is Naked juice so delicious but so expensive?!? I could not get past the horribly griay texture to enjoy the taste. Of $4.00 a boale I will skck with my Jack lalane Power Jucier & make my own. 14

15 SenKments Analysis - Example gritty cheaper ruins Protein Jamba Juice related to related to kind of part of posikve is a Texture Price sentiment Nutrition part of negakve related to Competitor Health/Wellness Morning is a Time walgreens kind of Pharmacy 15

16 RecommendaKons CollaboraKve Filtering Customer is looking for a product Receive 6ps: Receive personal offerings: 16

17 Brand Management 17

18 A Growing Trend ExpectaKons for BI are changing w/o anyone telling us Requirement Expecta6ons Reality Speed Speed of the Internet Speed = Infra + Arch + Design Accessibility Accessibility of a Smartphone BI Tool licenses & security Usability IPAD - Mobility Web Enabled BI Tool Availability Google Search Data & Report Metadata Delivery Speed of queskons Methodology & Signoff Data Access to everything Structured Data Scalability Cloud (Amazon) ExisKng Infrastructure Cost Cell phone or Free WIFI Millions 18

19 State of Data Today 19

20 Shaking The FoundaKons Old Way New Way 20

21 Business Users Understand Big Data Data needs validakon There is a lot of informakon that can be harnessed, if the right context is applied There is a lot of hidden meaning that can be applied to behavior with geo- spakal informakon to create in- depth informakon Data needs complexity abstrackon There are answers to queskons that are hidden in complex layers of informakon, and can be leveraged to provide accuracy in analykcal models All of these can be implemented successfully by only a team that has business users with expert knowledge across systems within the organizakon 21

22 Business Users Big Data Success Just providing data to Business Users does not guarantee success with Big Data This journey is a maturity model and needs to be understood from ExecuKve to Data Science Teams with the same context, business metadata and relevancy Business Users can write the business case but need the Data Science team to be available for providing the insights and associated success 22

23 Business Case The Need Market CompeKKon New Services New Products New Teams Customers Revenue Profitability AnalyKcs Campaign Success Social Media The Requirements New Data External Data Third Party Data Social Media Data Business Rules TransformaKon Rules Metadata SemanKc Libraries KPI s AnalyKcs VisualizaKon The Outcomes Business Insights CompeKKve AnalyKcs Market AnalyKcs Customer Behavior AnalyKcs Campaign AnalyKcs Customer Advocacy AnalyKcs Revenue AnalyKcs Forecast AnalyKcs PrescripKve AnalyKcs VisualizaKon 23

24 Business Case Skills Budget Stewardship Sponsorship 24

25 Design Key design steps IdenKfy the data IdenKfy the taxonomies required for the data Classify the data Categorize the data Design the metadata rules Design the business rules to process the data Design the excepkon rules Design the storage and implementakon infrastructure Where is the data to be stored How much storage will be needed What is the lifecycle of the data 25

26 SoluKon Architecture Transactional Databases Analytical Databases Semi Structured & Unstructured Data ERP / SCM D A T A I N T E G R A T I O N Enterprise Data Warehouse ( RDBMS or Columnar DB Plus HADOOP) S E M A N T I C L A Y E R Analytics Reports Portals 26

27 SemanKc IntegraKon Perform enkty extrackon from unstructured texts using advanced computakonal linguiskcs and natural language processing Organize unstructured data into meaningful, relevant and quankfiable knowledge automakcally Building the data graph of the associakons / clusters via semankc representakons of content Connects people with relevant informakon without the need for human supervision IdenKfy People, products, promokons, compektors, RelaKonship to each focused area and categories, products, authors, domains, influencers, and conversakon flow Support semankc reasoning and semankc query Display context through interackve visualizakon / hyper- graphs / theme clouds Help surface key word to refine searches Aid in categorizakon and discovery of new areas for research and analysis Link unstructured data to enterprise structure data via semankc transformakon 27

28 Text AnalyKcs Example I was involved in a car accident a few years ago, it took only one call to get everything going with ABC Company, the adjuster took great care of me, the car was repaired perfectly, and they covered everything. My biggest issue with XYZ company is their adjusters are as good and even beaer than ABC, but their penalty rates are so bad that when my car was in a fender bender and it was not my fault, they just raised my rate and also impacted my credit history. I have switched back to ABC and will go to BBB about XTZ. XYZ Company Used car New car Auto loan ABC Company related to related to kind of part of is a Competitor buying posikve sentiment Bank negakve part of related to Competitor Car Insurance Credit Card Resell is a Dealer Crash kind of Accident Classifier Parts of Speech Entity Extraction Semantic Relationship 28

29 Business Case To implement a Big Data pladorm or framework we need to develop a business case. The business case should include requirements, data, pladorms, approach, outcomes and skills. The business case should also include execukve sponsorship, governance and risk strategies. Once a good business case has been developed, it leads to pulling together the roadmap that will be the guideline for implementakon and measurement of success. By just creakng a business case you do not get successful, you need to understand the data and its requirements from a discovery and processing perspeckve. Business users, IT staff and new skills of StaKsKcs & Math will form a team of data scienksts and they can be mulkple within the organizakon. This team needs to be able to drive the data discovery process and idenkficakon of insights. 29

30 Program Challenges Enterprise inikakves change with strategy every two years at a minimum ERP, SCM and CRM packaged applicakons are Ked to technology choices and may serve negakve interest Business environment changes will need users to process different data than current or prior years Internal culture of the organizakon might not be a collaborakve environment Time to market or create a solukon and missed opportunity costs Chaos due to organizakon changes including key senior leadership 30

31 Business Challenge Gezng started with definikon, common objeckves and priorikes Funding and Sponsorship Challenges within the exiskng data architecture and technical solukon Data Architecture is inflexible across departments, divisions or companies Lack of governance data and program Lack of metadata program Stewardship 31

32 The Future Enterprise Enterprise AnalyKcs / ReporKng / Mashup / KPI Dashboards Enterprise Data Repository (Lake, Hub, Swamp & More: Big Data Pladorms) EDW (RDBMS / DBMS) Third Party Feeds Legacy Audio Video Image Graph OLTP ODS 32

33 ROADMAP 33

34 Value Big Data Roadmap Big Data & Roadmap What do We WANT to happen! Trigger a set of actions with a high degree of accomplishing it What IS Happening! WHAT Happened? Report on Yesterday WHY did it happen? Search for patterns WHAT will happen? Predict a ROI based on patterns A streaming dashboard providing insights into what is happening Pre- defined ReporKng Ad Hoc Queries PredicKve AnalyKcs DescripKve AnalyKcs PrescripKve AnalyKcs 34

35 Why Roadmap Create and Define a common vision for using informakon to run the business. Aligns business units to collaborate on real opportunikes to leverage informakon across departments, divisions and the enterprise. Leverages experkse and experience that has helped companies develop the right plan for the situakon. Protects your past and future investments through comprehensive analysis. Provides conclusive, pragmakc recommendakons and ackonable plans Establishes a consistent set of priorikes across business departments and divisions. Provides clear direckon to the IT team. Provides a clear cost/benefit rakonalizakon for planned inikakves. Very quick and cost effeckve process. 35

36 Roadmap Components Scope Goals Data Technology Projects Steering Commiaees Governance Skills Budget Rollout & Maintenance 36

37 Roadmap Deliverables Technology Reference Architecture Data Reference Architecture Sample Business model mapping to reference architecture Governance models Metadata Requirements / Readiness MDM Requirements / Readiness Skills & Readiness

38 Roadmap Outcomes As we see from this seckon, the roadmap process helps us in the following techniques Confirm Requirements Validate New Data Define TransformaKons & Business Rules Secure ExecuKve Sponsorship & Budgets Govern PrioriKes Define Skills & Resource Requirements Define KPI s Define Reports & Dashboards Plan the overall implementakon 38

39 EXAMPLES 39

40 Kraft - innovate with Kraft 40

41 My Starbucks Idea -Shaping the future of Starbucks 41

42 Fluevog - open shoe design Reference - hap:// open- innovakon- crowdsourcing- examples/ 42

43 Thinking Machines 43

44 Q&A 44

45 @datagenius Krish Krishnan Sixth Sense Advisors Inc. THANK YOU 45