Splunk for Data Science

Transcription

1 Copyright 2014 Splunk Inc. Splunk for Data Science Tom LaGa=a Data Splunk Olivier de Garrigues Sr Prof Services Consultant, Splunk

2 Disclaimer During the course of this we may make forward- looking statements regarding future events or the expected performance of the company. We you that such statements reflect our current and based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this are being made as of and date of its live If reviewed arer its live this may not contain current or accurate We do not assume any to update any forward- looking statements we may make. In any about our roadmap outlines our general product and is subject to change at without It is for purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no either to develop the features or described or to include any such feature or in a future release. 2

3 3 Key Takeaways Data Science is about extrac:ng ac:onable insights from data. Splunk is great for doing Data Science! Splunk complements other tools in the Data Science toolkit. 3

4 About Us! Tom LaGaAa, Data Scien:st Tom joined Splunk in Spring 2014 as a Data Scien@st specializing in Probability and Sta@s@cs. Tom is an expert on the mathema@cs of inference, and he enjoys func@onal programming in languages like Clojure, Haskell & R. At Splunk, Tom is helping to develop our internal and external Data Science program and curriculum. Tom has a Ph.D. in Mathema@cs from the University of Arizona, and un@l recently was a Courant Instructor at the Courant Ins@tute at New York University. Tom is based in New York City.! Olivier de Garrigues, Senior Professional Services Consultant Olivier is based in London on the EMEA Professional Services team and has helped out more than 40 customers in 10 countries on various Splunk projects in the past year and a half. Prior to this, he worked as a quan@ta@ve analyst with extensive use of MATLAB and R. He developed a keen interest in machine learning and enjoys dreaming about how to make Splunk be=er for data scien@sts, and helped develop the R Project App. Olivier holds an MS in Mathema@cs of Finance from Columbia University. 4

5 Splunk for Data Science

6 What is Data Science? Data Science is about insights from data.! Helps people make be=er decisions! Can be used for automated decision- making! Data Science is cross- and blends techniques & theories from: CS / Programming Math and Sta@s@cs Machine Learning Data Mining / Databases Data Visualiza@on! Don t be afraid of Data Science! Substan@ve / Domain Exper@se Social Science Communica@on and Presenta@on Accoun@ng, Finance and KPIs Business Analy@cs 6

7 Data Science & Teams There is no one size fits all data Data Science & teams are made up of people with complementary skill sets. Source: Schu= & O Neil. Doing Data Science

8 Splunk for Data Science Splunk is great for doing Data Science!! Integrate, query & visualize all the data: Plalorm for machine data Connects with any other data source! Easy- to- use Powerful algorithms out- of- the- box! Sharp and dashboards! Deliver results to both IT & Business users! Complements other Data Science tools (next slide) 8

9 Splunk and Data Science Tools Splunk complements other tools in the Data Science toolkit:! Hadoop: the workhorse of the Data Science world. Using Hunk, you can integrate Hadoop & HDFS seamlessly into Splunk.! R & Python: the preferred languages of Data Science. Execute R & Python scripts in your Splunk queries using the R Project App & SDK for Python! SQL & other RDBMS: valuable stores for customer & product data. Use Splunk s DB Connect App to mash rela@onal data up with machine data.! External tools: export finalized data from Splunk using the ODBC Driver Tip: do all your data processing in Splunk/Hunk, and export only the final results! D3 Custom Visualiza@ons: sharp dashboards & reports using Splunk 9

10 Splunk and Data Science Use Cases Splunk is a powerful tool for lots of Data Science use cases: Green Use Cases (easy out of the box) Yellow Use Cases Trend Forecas@ng D3 Custom Visualiza@ons A/B Tes@ng Predic@ve Modeling Root Cause Analysis Sen@ment Analysis Anomaly Detec@on Conversion Funnel/Pathing Market Segmenta@on More Algorithms via R & Python Topic Modeling Capacity Planning Correlate Data from 2+ Sources Data Munging & Normaliza@on KPIs & Execu@ve Dashboards 10

11 Data Science Use Cases

12 Use Case: Trend Trend given past & data, predict future values & events.! Common Forecast revenue & other KPIs Web server traffic & product downloads Customer conversion rates MTTR & server outages Resource & capacity planning (AWS App) Security threats (Enterprise Security App)! The true course of events can (and will) take only one of many divergent paths. But which one?! Be mindful of rare events & black swans! 12

13 predict command: forecast future trajectories series! Implements a Kalman filter to iden@fy seasonal trends! Gives an uncertainty envelope as a buffer around the trend! Tip: Always run the predict command on LOTS of past data. Capture low- frequency and high- frequency trends Splunk Solu@on: predict!! Remember: the future is always uncertain 13

14 Splunk Predict App David Carasso s Predict App: forecast future values of individual events. 8 minute walkthrough: h=ps:// Implements a Naïve Bayes classifier! You have to train models!! Train a model to predict any target field using any reference field(s): fields ref1, ref2,..., target train my_model from target!! Guess target field for incoming events: guess my_model into target!! Temporal or non- temporal predic@on (include _@me among reference fields) 14

15 Concept: Supervised Learning & Supervised learning: use observed training data to classify values of unknown tes1ng data! predict command (Kalman filter): Training data of past & real@me values. Tes@ng data range for future values! Predict App (Naïve Bayes classifier): Training data = events with reference & target fields. Tes@ng data = events with reference fields but not target field! Tip: only deploy models & algorithms a2er extensive tes@ng & evalua@on! More powerful learning algorithms using R Project App or SDK for Python 15

16 Demo: Predict App! Train a model to predict movie Ra@ng based on MovieID, UserID, Genre, Tag index=movielens Timestamp < UserID=593* eval original_rating = case(rating<3,"dislike", Rating=3,"Neutral", Rating>3,"Like") fields original_rating MovieID UserID Genre Tag train rating_model from original_rating!! Guess Ra@ng for test data based on trained model index=movielens Timestamp > UserID=593* guess rating_model into guessed_rating top original_rating guessed_rating!! Accuracy of model: correct on 97.6% of values! Tip: always train on LOTS of training data! Evaluate before deploying 16

17 Use Case: Analysis Analysis: the assignment of labels to textual data! Can be simple +1 vs. - 1, or more sophis@cated: happy, angry, sad, etc.! Analyze tweets, s, news ar@cles, logs or any other textual data! Social data correlates with other factors! Typically done via supervised learning: Train a model on labeled corpus of text Test the model on incoming text data! Read more about Sen@ment Analysis: Chapter 14 of Big Data Analy1cs Using Splunk (pp ) Michael Wilde & David Carasso. Social Media & Sen1ment Analysis..conf2012 r=.79 17% 1.8% 10% 36% 19% 3 rd 8 th 4 th 1 st 2 nd 2011 Irish General Elec@on 17

18 Splunk Analysis App David Carasso s Sen@ment Analysis App assigns binary sen@ment values to textual data (logs, tweets, , etc.)! Naïve Bayes classifier under the hood! Twi=er & IMDB models out of the box! Can guess language of authorship, and heat, a measure of emo@onal charge! Tip: compare rela@ve sen@ment changes & groups! How to train your own models: h=p://answers.splunk.com/answers/

19 Demo: Analysis App 19

20 Use Case: Anomaly An anomaly (or outlier) is an event which is vastly dissimilar to other events! Anomaly is one of Splunk s most common use cases. Examples: Transac@ons which occur faster than humanly possible DDoS a=acks from IP address ranges High- value customer purchase pa=erns! Quick techniques for finding sta@s@cal outliers: Non- average outliers: more than 2*stdev from the avg Non- typical outliers: more than 1.5*IQR above perc75 or below perc25! Tip: save these as even=ypes for automated outlier detec@on! Once anomalies have been found, dig deeper to discover root causes 20

21 Splunk cluster! Anomalies are dissimilar to other events (by We can use clustering algorithms to help us detect anomalies: Non- anomalous events typically form a few large clusters Anomalous events typically form lots of small clusters! Cluster your data, sort ascending: cluster showcount=true labelonly=true sort cluster_count cluster_label!! Remember: there is no right way to find all anomalies. Explore your data! 21

22 Concept: Unsupervised Learning & Clustering! A clustering algorithm is any process which groups together similar things (events, people, etc), and separates dissimilar things (events, people, etc.)! Clustering is unsupervised: choose labels based on pa=erns in the data! Clustering is in the eye of the beholder: Lots of different clustering algorithms Lots of different similarity func@ons! Do not confuse with: Computer cluster: a group of computers working together as a single system Splunk cluster: a group of Splunk indexers replica@ng indexes & external data 22

23 Demo: cluster! 23

24 Splunk Other Commands! anomalies: Assigns an unexpectedness score to each event! anomalousvalue: Assigns an anomaly score to events with anomalous values! outlier: Removes or truncates outliers! kmeans: Powerful clustering algorithm. You choose k = # of clusters 24

25 Splunk Prelert (Partner App)! Manages Anomaly directly Pre- built dashboards, alerts, API. Use cases: Security, IT Ops / APM, DevOps Godfrey Sullivan: "beau@fully adjacent and complimentary to what Splunk does! Can download from Splunk Apps May save with Anomaly Detec@on Can also be good source of inspira@on for your own Anomaly Detec@on dashboards! Keep in mind Prelert is a paid app: Cost: 5GB 25

26 Use Case: Market Market group customers according to common needs and and develop strategies to target them Market segments are internally homogeneous, and externally heterogeneous i.e., market segments are clusters of customers! Many reasons for Market Different market segments require different strategies Customers in same segment have similar product preferences. Different segments, different preferences Segments should be reasonably stable, to allow for historical analysis (good for Data Science)! Use Splunk s clustering algorithms to iden@fy and label market segments! 26

27 Data

28 Intro to Data Data is the and study of the visual of data, and is a vital part of Data Science The goal of data visualiza@on is to communicate informa1on: Visualiza@ons communicate complex ideas with clarity, precision, and efficiency Transmission speed of the op@c nerve is about 9Mb/sec fast image processing Pa=ern matching, edge detec@on Visualiza@ons pack lots of informa@on into small spaces. More than text alone! 28

29 Telling Stories with Data We process data in linear even dashboards go top- to- bo=om! help pierce the monotony of text, number & data streams Think about the story you re telling:! Empathize with the viewer What s their takeaway?! A good visualiza@on tells its own story: Island Na@on Obtains Favourable Balance of Trade; Goes On To Rule The World! Weave mul@ple visualiza@ons together to tell more effec@ve stories 29 William Playfair (1786)

30 Splunk Source: New York Times. May 17,

31 Splunk Source: New York Times. May 17,

32 Tips for Data Plot the most important keys on x & y axes You choose most important. You might need >1 visualiza@on.! Manipulate size, color and shape to convey addi@onal informa@on! Annotate, label and add icons! Use chart overlay to correlate data sources. Mix histograms & line charts! Manipulate numerical scale: linear vs. log scales (previous 2 slides)! Read more about Data Visualiza@on: Tableau s whitepaper, Visual Analysis Best Prac1ces (2013) Edward TuRe s The Visual Display of Quan1ta1ve Informa1on (2001) 32

33 D3 Custom in Splunk! Splunk now supports D3 with some minor Satoshi s talk: I want that cool viz in Splunk!! Resources for Custom Visualiza@ons: Splunk Web Framework Toolkit h=ps://apps.splunk.com/app/1613/ Splunk 6.x Dashboard Examples h=ps://apps.splunk.com/app/1603/ Custom SimpleXML Extensions h=p://apps.splunk.com/app/1772/ Lots more D3 visualiza@ons for h=ps://github.com/mbostock/d3/wiki/gallery 33

34 Demo: Sankey Chart 34

35 How- to for Sankey Charts! Install the Custom SimpleXML Extensions app: h=p://apps.splunk.com/app/1772/! Create your own app, and install Sankey chart components: Drop autodiscover.js in Copy & paste /sankeychart/ subfolder into $SPLUNK_HOME/etc/apps/<YOURAPP>/ Restart Splunk! In your dashboard: Include script="autodiscover.js" in <form> or <dashboard> opening tag Insert XML snippet from 2- or 3- node Sankey dashboard example Change 2 instances of custom_simplexml_extensions to <YOURAPP> Update search and data- op@ons parameters (nodes) in XML to reflect your data 35

36 Know Your Audience! Finally, keep in mind your audience: who are they, what do they care about, and how do they want to consume the data? KPIs, charts, tables with icons Analyst: KPIs & metrics. Sharp images for their own reports & decks. Tableau Data output clean data to organized data stores (Hunk, HDFS, SQL, NoSQL) Sysadmin: sparklines, gauges for & MTTR, tables with highlighted anomalies Security Ops: maps with detailed overlays, drill down on anomalous events.! Bring it back to the business problem & use 36

37 3 Key Takeaways Data Science is about extrac:ng ac:onable insights from data. Splunk is great for doing Data Science! Splunk complements other tools in the Data Science toolkit. 37

38 List of References Good books on Data Science:! Schu= & O Neil. Doing Data Science. O Reilly 2013! Provost & Fawce=. Data Science for Business. O Reilly 2013! Max Shron. Thinking With Data. O Reilly 2014! Edward TuRe. The Visual Display of Quan1ta1ve Informa1on. Graphics Press 2001! Zumel & Mount. Prac1cal Data Science with R. Manning 2014! Has@e et al. Elements of Sta1s1cal Learning. Springer- Verlag 2009 (free PDF!) Using Splunk for Data Science:! Zadrozny, Kodali (and Stout). Big Data Analy1cs Using Splunk. Apress 2013! David Carasso. Exploring Splunk. CITO Research 2012! David Carasso. Data Mining with Splunk..conf2012! Michael Wilde & David Carasso. Social Media & Sen1ment Analysis..conf2012 Good free references:! Tableau. Visual Analysis Best Prac1ces. Tableau 2013! King & Magoulas Data Science Salary Survey. O Reilly 2013! DJ Pa@l. Building Data Science Teams. O Reilly 2013! Cathy O Neil. On Being A Data Skep1c. O Reilly

39 THANK YOU