10 BEST PRACTICES FOR SOLUTION ARCHITECTURES THAT WOULD TAME BIG DATA!!!

Transcription

1

2 10 BEST PRACTICES FOR SOLUTION ARCHITECTURES THAT WOULD TAME BIG DATA!!!

3 BIG DATA BEST PRACTICE-1 USE CASE! USE CASE! USE CASE ( FRAME IT TIGHT)

4 THE IDEA IN BRIEF What are the questions at the heart of the problem? Formulate the hypothesis/questions at the heart of the issue! Distill them into a clear set of hypothesis to be tested Remember Hadoop and associated technology components are a means Isolate $ Denting Analytical Use Case

5 REAL LIFE EXAMPLE : CURATING USE CASE IN TELECOM SECURITY INTELLIGENCE Business Context What new signals to listen to prevent adverse events from happening? 4 Data Pools Netsweepeer logs Radius logs Switch CDR MMS logs 2 Use cases Watch list analysis + Network link analysis MMS Video virality

6 Have an intensive ½ day cross functional workshop with business to boil down the game changing use case Is it a nice to have use case or a $ impacting use case? Who is the consumer of the use case? How does it help him optimize cost or reduce risk or increase revenue? Business backwards and NOT technology forward

7 BIG DATA BEST PRACTICE-2 IMPACT AHA MOMENT IN DAYS. START WITH A SKELETAL WORKING SOLUTION ( MVP )

8 THE IDEA IN BRIEF DELIVER FIRST BIG DATA AHA MOMENT IN DAYS Skeletal MVP : End to end implementation that links all architectural components together Could be the answer to a previously unanswered question Propels momentum of Big data project

9 A REAL LIFE EXAMPLE Industry = OTA Context : Important to improve look to book Is there a co-relation between response time of a web page and the look to book ratio? Hadoop cluster + Infobright + Hive jobs ready in 3 weeks Scaled data and improvised dashboard experience for another 3 weeks Business readout in 6 weeks

10 THEREFORE Break it into 3 chunks 30 day milestones 60 day milestones 90 day milestones In 30 days plan to cover functional breadth Hadoop infrastructure + cluster Integrate disparate components data pipeline, Columnar database, machine learning process, Hadoop cluster Have a small file go from start to end thru the process chain In 60 days plan to cover scalability Scale for 12 months data atleast Tableau / Pentaho In 90 days plan to cover bells n whistles Configurators Alerters Additional abtraction Don t wait for 6-9 months!

11 BIG DATA BEST PRACTICE-3 ACTIONS NOT INSIGHTS ACTION INSIGHTS DATA

12 BEST PRACTICE-3 ACTIONS NOT INSIGHTS Actions are executed in the frontline Call centre Mobile Store channel Digital channel Actions could be Behaviour based discounts Help close a digital transaction Serve customized webpage Take proactive actions Insights are nice to know Actions impact $

13 THEREFORE WHAT ACTIONS ARE DRIVEN AS A RESULT OF THESE INSIGHTS? HOW ARE WE DISSEMINATING INSIGHTS TO FRONT LINE CHANNELS? ASK SO WHAT 5 TIMES!!!

14 BIG DATA BEST PRACTICE-4 : LISTEN TO UNSTRUCTURED INTELLIGENCE FOR STRONG SIGNALS

15 REAL LIFE EXAMPLE Keyword frequency Leaks, Leakage, Noise, Sound, Vibrations Noise / leakage frequency is a better predictor of repeat sales than any other indicators including marketing spends!!!

16 A REAL LIFE EXAMPLE Business Question How can we create a strategy to respond to what we are hearing about XYZs buzz online? Raw data Statistical Technique Text mining Visual data exploration Hypothesis testing Affinity analysis Slide 16 XYZ Online Buzz analysis Insights derived Sentiment trends :+/- Sentiment benchmark with McDonalds Top keywords for XYZ Top keywords for McDonalds Keyword affinities Business Action Theme specific campaigns NPD process Instore experience Reverse impact of negative buzz

17 WHERE DO CUSTOMERS EXPRESS THEMSELVES? 136 posts 552 posts 2854 posts Yelp.com Epinions.com planetfeedback.com 1500 posts 500 posts Twitter.com Facebook.com Universe of XYZ sentiment data = 5 sources, 5556 posts,3 years data we s phase-1 analysis = posts, 2 years data

18 SOURCE = TWITTER.COM

19 SOURCE = YELP.COM

20 SOURCE = FACEBOOK.COM

21 STEP BY STEP SENTIMENT TEXT MINING PROCESS Process Blogs Customer review sites Online consumer forum Customers\Ven dors s Unstructured data from Applications Input Inferences Customer s sentiments Output

22 OVERALL SENTIMENTS DASHBOARD

23 THEREFORE R text mining algorithm RHadoop

24 Which devices are infected from a malicious attack? BIG DATA BEST PRACTICE-5 : COLUMNAR &IN MEMORY ARCHITECTURES TO SPEED UP CHAIN OF THOUGHT

25 HOW TO HANDLE NEEDLE IN A HAYSTACK WORKLOADS? What happened on firewall-3 between 3:17 and 3:21 am? How many payment gateway drops happened between 9:47 am and 9:52 am on 15-Nov-2012? Data forensic queries supporting chain of thoughts

26 Columnar DB Concept in Brief Id Name Designation Tenure S1 Prem Founder 8 S2 Simon Security Architect 5 S3 Bhavana Sales Head 6 S4 Ram CEO 3 S5 Shyam Developer 1 S1PremFounder8 S2SimonSecurityArchitect5 S3BhavanaSalesHead6 S4RamCEO3 S5ShyamDeveloper1 S1S2S3S4S5PremSimonBhavanaRamShyamFounderSecurityHeadSalesHeadCEODeveloper

27 IN MEMORY DATABASES! interactive or real-time query for large datasets =key to analyst productivity (support chain of thought analysis). Chain of thought analysis = Explore data torrent by quickly running off a series of iterative queries, each informed by the last. Most solutions aren t fast enough and reduce analytical effectiveness when users chain of thought process is interrupted In memoy DB Tools Dremel at Google, Druid at Metamarkets, Sting at Netflix, Cloudera s Impala C Berkeley s AMPLab s Spark, SAP Hana, Platfora.

28 THEREFORE Examine columnar databases and inmemory databases to speed up important query workloads Download evaluation version of Actian, Infobright and do a POC

29 BEST PRACTICE-6 HOW TO PLAN FOR 100 X SCALABILITY? BIG DATA BEST PRACTICE-6 : THINK 100 X SCALABILITY!!!

30 REAL LIFE EXAMPLE Industry = Telecom Business context National content filtering solution Events Generated Per Day : 1 Billion Events New URL s Classified per Day : 1 Million Daily log Volume : 400Gb average

31 The data torrent The Organisation Price sensitive search Ratings based ordering Real time sense making Comparator events Store search Payment Gateway events Basket add events BIG DATA BEST PRACTICE-7 : DETECT DATA PATTERNS IN REAL TIME!!!

32 THE CONTEXT Velocity is high Decision making window is low Cost of not intervening is high

33 REAL TIME EXAMPLE Decision window = 8 mins If a high value customer ( decile = 1 on last 36 months revenue ) and intra book interval > threshold and recency of search < 70 then route to call center channel

34 THEREFORE Include S4 and other real time analytics into your Big data reference architecture

35 BIG DATA BEST PRACTICE-8 CAPTOLOGY = PERSUASION THRU TECHNOLOGY

36 THE BASICS Captology = Persuasion thru technology DESIGN FOR BEHAVIOURAL CHANGE Persuasion examples Users to change channel behaviour ( Move from Desktop to Mobile channel ) Persuade users to advocate friends

37 CAPTOLOGY IN ACTION Captology in Insurance Reduce rates each time a person reports his or her exercise behaviour to a group of peers online Captology in Social

38 THERE ARE TOO MANY GOOD PRODUCTS HIDDEN BEHIND BAD USER INTERFACES PRODUCT = INTERFACE FOR BIZ USER, WHAT LIES UNDER THE HOOD DOES NOT MATTER

39 BIG DATA BEST PRACTICE-9 STRETCH KEY BIG DATA COMPONENTS TO SEE WHAT BREAKS!

40 BEST PRACTICE-9 INTERSECT OF MOVING PARTS ARE THE WEAK LINKS Big Data Moving moving parts Columnar databases Hadoop clusters Advanced visualisation layer Real time components Data pipelines API s scrappers to syndicate info Bridge to existing DW Hadoop Cluster The intersect can give away as data / user volumes increase A real life big data architecture architecure Event loggers Hbase/Cassandra for high velocity event absorption Sqoop/Flume for data ingestion Hadoop cluster for massive data crunching R for extracting patterns Columnar database for 10 x lightning retrieval Tableau for advanced visualisation S4 for real time analytics Channel integration components Infobright Columnar DB R Predictor ranking

41 THEREFORE WATCH THE FOLLOWING 4 WEAK LINKS 1. Link between Operational event streams and Hadoop cluster 2. Link between Hadoop cluster and Columnar database 3. Link between Columnar database and the visualisation tool 4. Time it takes for the machine learning algorithm to run

42 HIGH VELOCITY DATA PIPELINE WHAT'S THE INGESTION RATE IN EVENTS PER SECOND OF THE DATA PIPELINE?

43 BIG DATA BEST PRACTICE-10 EMBED MACHINE LEARNING PROCESSES TO DETECT PATTERNS

44 BEST PRACTICE-10 : 7 CORE MACHINE LEARNING BUILDING BLOCKS FOR ORCHESTRATING ANALYTICAL PROCESSES Collaborative filtering A/B testing Apriori Clustering Text mining Scoring models Optimization

45 TO SUMMARIZE THE 10 CORE BIG DATA BEST PRACTICES!

46 THANK YOU! QUESTIONS? COMMENTS? THOUGHTS?