Predictive Analytics Techniques: What to Use For Your Big Data. March 26, 2014 Fern Halper, PhD

Transcription

1 Predictive Analytics Techniques: What to Use For Your Big Data March 26, 2014 Fern Halper, PhD

2 Presenter

3 Proven Performance Since 1995 TDWI helps business and IT professionals gain insight about data warehousing, BI, and analytics: High-quality, vendor-neutral educational offerings Independent analyst research staff and thought leadership Trusted sources of emerging information and trends Ability to bring together qualified BI/DW professionals and solution providers Premium Membership, conferences, seminars, research, publications, topical portals, whitepaper library, and numerous online programs 3

4 Agenda Introduction to big data and predictive analytics Popular predictive analytics methodologies Examples Guidelines Deployment models

5 Big Data M2M/IoT Volume Mobile/Location Social Text Formats 5

6 A Confusion of Words

7 Big Data Analytics Text analy,cs Stream mining Predic,ve analy,cs Link analysis Big Data Analy,cs Slicing and dicing Visual discovery Etc.

8 Predictive Analytics A statistical or data mining solution consisting of algorithms and techniques that can be used on both structured and unstructured data to determine outcomes

9 A Lot of It Is Used to Predict Behavior People Churn Marketing Fraud detection Machine Operations maintenance And much, much more! Good source for use cases

10 Of Course, It Isn t Just About Modeling CRISP Lifecycle

11 A Vast Array of Techniques Source: TDWI BPR on Predictive Analytics, 2014; n=242

12 Supervised Use it when you know outcomes of interest Leave vs. stay Revenue prediction Need enough data for training, testing, validation

13 Unsupervised Does not include target information Looks for commonalities/hidden structures in data May not produce useful insight Is it prediction?

14 Techniques Supervised Classification Regression Neural networks Unsupervised Clustering Association Supervised Deep learning, auto-encoders Decision trees, random forests, gradient boosting Support vector machines, Bayesian classifiers, principal component, discriminant analysis Unsupervised Nearest-neighbor mapping, k-means clustering, selforganizing maps Factor analysis, link analysis

15 Decision Trees Good for classification and prediction with known, discrete outcomes

16 Linear Regression Used to predict a continuous variable from independent variables

17 Artificial Neural Networks (1) Biological to Mathematical Source agh.edu

18 Artificial Neural Networks (2) Can be used on a range of problems; good for classification and estimation Source: Commonsenseatheism.com

19 Clustering Used to group observa,ons by perceived similarity Source: Babelomics

20 Association Rule Mining Transac'on Items 1 milk, leduce 2 leduce, diapers, beer, cookies 3 milk, diapers, beer, plas,c bags 4 leduce, milk, diapers, beer 5 leduce, milk, diapers, plas,c bags Diapers -> Beer Used to find relationships Two concepts: support and confidence

21 Quick Quiz How much revenue will this customer bring? Regression Who is going to take a certain action? Classification What are my customer segments? Clustering If a customer buys X, what else might it buy? Association rules

22 Strengths & Weaknesses: Decision Trees Strengths Easy to understand Rules vs. equations Easy to explain Not a black box Data doesn t have to follow any distribution Can handle interactions between variables Weaknesses Continuous value predictions Can be computationally expensive to train Can have problems if many classes and few training examples Overfitting

23 Strengths & Weaknesses: Regression Strengths Simple to use Easy to explain through independent variables Weaknesses Relationship needs to be linear Hard-to-handle categorical variables or variables that interact Outliers hard to model

24 Strengths & Weaknesses: Neural Networks Strengths Good for a specific class of problems May be easy to implement Non-linear/interaction variables Weaknesses Hard-to-explain output (black box) Output might be unpredictable Training can take a long time

25 Strengths & Weaknesses: K Means Strengths Good for large datasets Simple Efficient Weaknesses Need to specify K upfront Sensitive to outliers, which may result in incorrect cluster boundaries Needs a mean (categorical data?)

26 Strengths & Weaknesses: Association Rules Strengths Simple Text data (categorical) Weaknesses Can be computationally expensive Potential for spurious patterns Rules do not mean causality

27 Ensemble Modeling Multiple models are combined to solve a problem

28 Vendors Are Offering a Range of Options for Predictive Analytics UI easier to use: visual vs. code based Automation Collaboration/interactivity Cloud options Operationalizing and embedding advanced analytics

29 Operationalizing An example: 29

30 TDWI Big Data Maturity Model

31 QUESTIONS?