# Bayesian Ensemble Learning for Big Data

 To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video
Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Bayesian Ensemble Learning for Big Data Rob McCulloch University of Chicago, Booth School of Business DSI, November 17, 2013 P: Parallel

2 Outline: (i) ensemble methods. (ii) : a Bayesian ensemble method. P: Parallel (iii) and big data : - Parallel version of. - and.

3 Four papers: Regression, Annals of Applied Statistics, (Chipman, George, McCulloch) Parallel Regression, Journal of Computational and Graphical Statistics, forthcoming. (M. T. Pratola, H. Chipman, J.R. Gattiker, D.M. Higdon, R. McCulloch and W. Rust) Bayes and Big Data: The Consensus Monte Carlo Algorithm, submitted (Steven L. Scott, Alexander W. Blocker,Fernando V. Bonassi, Hugh A. Chipman, Edward I. George, and Robert E. McCulloch) P: Parallel Chipman: Acadia; George: U Penn, Wharton; Pratola: Ohio State. Gattiker, Hidgon, Rust: Los Alamos; Scott, Blocker, Bonassi: Google. And, Reversal of fortune: a statistical analysis of penalty calls in the National Hockey League. (Jason Abrevaya, Robert McCulloch) Abrevaya: University of Texas at Austin.

4 The Hockey Data Glen Healey, commenting on an NHL broadcast: Referees are predictable. The flames have had three penalties, I guarantee you the oilers will have three. P: Parallel Well, guarantee seems a bit strong, but there is something to it. How predictable are referees?

5 Got data on every penalty in every (regular season) game for 7 seasons around the time they switched from one referee to two. For each penalty (after the first one in a game) let revcall = 1 if current penalty and previous penalty are on different teams, P: Parallel 0 otherwise. You know a penalty has just been called, which team is it on? is it a reverse call on the other team??? Mean of revcall is.6!

6 For every penalty (after the first one in a game) we have: Table: Variable Descriptions Variable Description Mean Min Max Dependent variable revcall 1 if current penalty and last penalty are on different teams Indicator-Variable Covariates ppgoal 1 if last penalty resulted in a power-play goal home 1 if last penalty was called on the home team inrow2 1 if last two penalties called on the same team inrow3 1 if last three penalties called on the same team inrow4 1 if last four penalties called on the same team tworef 1 if game is officiated by two referees Categorical-variable covariate season Season that game is played 1 7 P: Parallel

7 Table: Variable Descriptions Variable Description Mean Min Max Other covariates timeingame Time in the game (in minutes) dayofseason Number of days since season began numpen Number of penalties called so far (in the game) timebetpens Time (in minutes) since the last penalty call goaldiff Goals for last penalized team minus goals for opponent gf1 Goals/game scored by the last team penalized ga1 Goals/game allowed by the last team penalized pf1 Penalties/game committed by the last team penalized pa1 Penalties/game by opponents of the last team penalized gf2 Goals/game scored by other team (not just penalized) ga2 Goals/game allowed by other team pf2 Penalties/game committed by other team pa2 Penalties/game by opponents of other team P: Parallel n = 57, 883.

8 How is revcall related to the variables? inrow2=0 inrow2=1 revcall= revcall= P: Parallel inrow2=1: If the last two calls were on the same team then 64% of the time, the next call will reverse and be on the other team. inrow2=0: If the last two calls were on different teams, then the frequency of reversal is only 56%.

9 Of course, we want to relate revcall to all the other variables jointly! Well, we could just run a logit, but with all the info, can we, fairly automatically, get a better fit than a logit gives? P: Parallel What could we try?

10 Data Mining Certificates Online Stanford Center for Professional Development Data Mining and Analysis STATS202 Description In the Information Age, there is an unprecedented amount of data being collected and stored by banks, supermarkets, internet retailers, security services, etc. So, now that we have all this data, what do we with it? The discipline of data mining and analysis provides crunchers with the tools and framework to discover meaningful patterns in data sets of any size and scale. It allows us to turn all of this data into valuable, actionable information. In this course, learn how to explore, analyze, and leverage data. P: Parallel Topics Include Decision trees Neural networks Association rules Clustering Case-based methods Data visualization

11 Modern Applied Statistics: Data Mining STATS315B Online Description P: Parallel Examine new techniques for predictive and descriptive learning using concepts that bridge gaps among statistics, computer science, and artificial intelligence. This second sequence course emphasizes the statistical application of these areas and integration with standard statistical methodology. The differentiation of predictive and descriptive learning will be examined from varying statistical perspectives. Topics Include Classification & regression trees Multivariate adaptive regression splines Prototype & near-neighbor methods Neural networks Instructors Jerome Friedman, Professor Emeritus, Statistics

12 Advanced Analytics for Customer Intelligence Using SAS Predictive Modeling for Customer Intelligence: The KDD Process Model A Refresher on Data Preprocessing and Data Mining Advanced Sampling Schemes cross-validation (stratified, leave-one-out) bootstrapping Neural networks multilayer perceptrons (MLPs) MLP types (RBF, recurrent, etc.) weight learning (backpropagation, conjugate gradient, etc.) overfitting, early stopping, and weight regularization architecture selection (grid search, SNC, etc.) input selection (Hinton graphs, likelihood statistics, brute force, etc.) self organizing maps (SOMs) for unsupervised learning case study: SOMs for country corruption analysis P: Parallel Support Vector Machines (SVMs) linear programming the kernel trick and Mercer theorem SVMs for classification and regression multiclass SVMs (one versus one, one versus all coding) hyperparameter tuning using cross-validation methods case study: benchmarking SVM classifiers

13 Opening up the Neural Network and SVM Black Box rule extraction methods (pedagogical versus decompositional approaches such as neurorule, neurolinear, trepan, etc. two-stage models A Recap of Decision (C4.5, CART, CHAID) Regression splitting/stopping/assignment criteria P: Parallel bagging boosting stacking random forests

14 A Tree goaldiff < 0.5 inrow2 < 0.5 timebetpens < numpen < 2.5 timebetpens < inrow2 < 0.5 inrow3 < 0.5 P: Parallel no:0.35 yes:0.65 goaldiff < 0.5 tworef < Last penalized was not ahead Last two penalties on same team Not long since last call one ref 72% revcall. Last penalized was ahead it has been a while since last penalty last three calls not on same team 48% revcall.

15 : A single tree can be interpretable, but it does not give great in-sample fit or out-of-sample predictive performance. Ensemble methods combine fit from many trees to give an overall fit. P: Parallel They can work great!

16 Let s try Random Forests (Leo Brieman). My impression is that Random Forests is the most popular method. Build many (a forest) of very big trees, each of which would overfit on its own. Randomize the fitting, so the trees vary (eg. In choosing each decision rule, randomly sample a subset of variables to try). To predict, average (or vote) the result of all the trees in the forest. P: Parallel For example, build a thousand trees!!! Have to choose the number of trees in the forest and the randomization scheme. Wow! (crazy or brilliant?)

17 Let s try Random Forests and trees of various sizes. For Random Forests we have to choose the number of trees to use and the number of variables to sample. P: Parallel I ll use the default for number of variables and try 200,500, and 1500 trees in the forest.

18 We have 57,883 observations and a small number of variables so let s do a simple train-test split. train: use 47,883 observations to fit the models. P: Parallel test: use 10,000 out-of-sample observations to see how well we predict.

19 Loss for trees of different sizes, and 3 forests of different sizes (each forest has many big trees!). Smaller loss is better. Loss is measured by the deviance ( -2*log-likelihood (out-of-sample)). P: Parallel RF200 RF500 RF TREE5 TREE10 TREE15 TREE20 TREE25 TREE30 LOGIT model Logit does best! Let s try boosting and.

20 LOGIT GBM200 GBM300 GBM500 P: Parallel model GBM is Jerry Friedman s boosting. The number is the number of trees used. Other parameters left at default. is Regression. We used the default prior and 200 trees.

21 We want to fit the fundamental model: Y i = f (X i ) + ɛ i P: Parallel is a Markov Monte Carlo Method that draws from f (x, y) We can then use the draws as our inference for f.

22 To get the draws, we will have to: Put a prior on f. Specify a Markov chain whose stationary distribution is the posterior of f. P: Parallel

23 Simulate data from the model: Y i = x 3 i + ɛ i ɛ i N(0, σ 2 ) iid n = 100 sigma =.1 f = function(x) {x^3} set.seed(14) x = sort(2*runif(n)-1) y = f(x) + sigma*rnorm(n) xtest = seq(-1,1,by=.2) P: Parallel Here, xtest will be the out of sample x values at which we wish to infer f or make predictions.

24 y plot(x,y) points(xtest,rep(0,length(xtest)),col= red,pch=16) P: Parallel Red is xtest. x

25 library(bayestree) rb = bart(x,y,xtest) length(xtest) [1] 11 dim(rb\$yhat.test) [1] P: Parallel The (i, j) element of yhat.test is the i th draw of f evaluated at the j th value of xtest. 1,000 draws of f, each of which is evaluated at 11 xtest values.

26 plot(x,y) lines(xtest,xtest^3,col= blue ) lines(xtest,apply(rb\$yhat.test,2,mean),col= red ) qm = apply(rb\$yhat.test,2,quantile,probs=c(.05,.95)) lines(xtest,qm[1,],col= red,lty=2) lines(xtest,qm[2,],col= red,lty=2) y P: Parallel

27 Example: Out of Sample Prediction Did out of sample predictive comparisons on 42 data sets. (thanks to Wei-Yin Loh!!) p=3 65, n = 100 7, 000. for each data set 20 random splits into 5/6 train and 1/6 test use 5-fold cross-validation on train to pick hyperparameters (except -default!) gives 20*42 = 840 out-of-sample predictions, for each prediction, divide rmse of different methods by the smallest P: Parallel + each boxplots represents 840 predictions for a method means you are 20% worse than the best + -cv best + -default (use default prior) does amazingly well!! Rondom Forests Neural Net Boosting cv default

28 A Regression A Single Tree Regression Model Tree Model t T Let denote T denote the tree the structure cluding tree the structure decision including rules the decision rules. t M = {µ 1, µ 2, µ b } denote e set M of = bottom {µ 1, µ 2 node,..., µ's. µ b } denotes the set of t g(x;"), bottom" = node (T, M) µ s. be a gression tree function that signs Leta g(x; µ value T, M), to x be a regression tree function that assigns a µ value to x. x 5 < c x 2 < d x 2 % d µ 1 = -2 µ 2 = 5 x 5 % c µ 3 = 7 P: Parallel A Single Tree Model: Y = g(x;!) +! 7 A single tree model: y = g(x; T, M) + ɛ.

29 A coordinate view of g(x; T, M) The Coordinate View of g(x;") x 5 < c x 5 % c µ 3 = 7 x 5 c µ 3 = 7 P: Parallel x 2 < d x 2 % d µ 1 = -2 µ 2 = 5 µ 1 = -2 µ 2 = 5 d x 2 Easy to see that g(x; Easy to T see, M) that isg(x;") justis ajust step a step function. 8

30 The Ensemble Model The Model Let " = ((T 1,M 1 ), (T 2,M 2 ),, (T m,m m )) identify a set of m trees and their µ s. Y = g(x;t 1,M 1 ) + g(x;t 2,M 2 ) g(x;t m,m m ) +! z, z ~ N(0,1) P: Parallel µ 1 µ 4 µ 2 µ 3 E(Y x, ") is the sum of all the corresponding µ s at each tree bottom node. m = 200, 1000,..., big,.... Such a model combines additive and interaction effects. f (x ) is the sum of all the corresponding µ s at each bottomremark: node. We here assume! ~ N(0,! 2 ) for simplicity, but will later see a successful extension to a general DP process model. Such a model combines additive and interaction effects. 9

31 Complete the Model with a Regularization Prior π(θ) = π((t 1, M 1 ), (T 2, M 2 ),..., (T m, M m ), σ). π wants: Each T small. Each µ small. nice σ (smaller than least squares estimate). P: Parallel We refer to π as a regularization prior because it keeps the overall fit from getting too good. In addition, it keeps the contribution of each g(x; T i, M i ) model component small, each component is a weak learner.

32 MCMC Y = g(x;t 1,M 1 ) g(x;t m,m m ) + & z plus #((T 1,M 1 ),...(T m,m m ),&) Connections to Other Modeling Ideas First, it is a simple Gibbs sampler: ayesian Nonparametrics: Lots (T of parameters (to make model flexible) i, M i ) (T 1, M 1,..., T i 1, M i 1, T i+1, M i+1,..., T m, M m, σ) A strong prior to shrink towards simple structure (regularization) shrinks σ towards (T 1, Madditive 1,..., models..., T m with, Msome m ) interaction ynamic Random Basis: g(x;t 1,M 1 ),..., g(x;t m,m m ) are dimensionally adaptive To draw (T i, M i ) we subract the contributions of the radient other Boosting: trees from both sides to get a simple one-tree model. Overall fit becomes the cumulative effort of many weak learners We integrate out M to draw T and then draw M T. 12 P: Parallel

33 Simulating p(t data) with the Bayesian CART Algorithm To draw T we use a Metropolis-Hastings within Gibbs step. We use various moves, but the key is a birth-death step. Because p(t data) is available in closed form (up to a norming constant), we use a Metropolis-Hastings algorithm. Our proposal moves around tree space by proposing local modifications such as P: Parallel? => propose a more complex tree? => propose a simpler tree Such modifications are accepted according to their compatibility with p(t data).... as the MCMC runs, each tree in the sum will grow and shrink, swapping fit amongst them... 20

34 Build up the fit, by adding up tiny bits of fit.. P: Parallel

35 Nice things about : don t have to think about x s (compare: add xj 2 and use lasso). don t have to prespecify level of interaction (compare: boosting in R) competitive out-of-sample. stable MCMC. stochastic search. simple prior. uncertainty. small p and big n. P: Parallel

36 Back to Hockey How can I use to understand the hockey data?? I have a problem!!!. P: Parallel The sum of trees model is not interpretable. Other methods (neural nets, random forests,..) have the same problem.

37 Well, you can look into the trees for things you can understand: - what variables get used. - what variables get used together in the same tree. P: Parallel Or, estimate p(revcall = 1 x) at lots of interesting x and see what happens!!

38 In practice, people often make millions of predictions. We ll keep it simple for the hockey data: (i) Change one thing at a time (careful! you can t increase numpens without increasing time-in-game) (ii) Do a 2 5 factorial expermiment where we move 5 things around, with each thing having two levels. This will give us 32 x vectors to esimate p(revcall = 1 x) at. P: Parallel

39 Change the binary x s one at a time: p(revcall) P: Parallel tworef 0 tworef 1 ppgoal 0 ppgoal 1 home 0 home 1 inrow2 0 inrow2 1 inrow3 1 inrow4 1

40 Change the score: x is lead of last penalized team. p(revcall) P: Parallel

41 2 5 factorial: r or R: Last two penalties on different teams or the same team. g or G: Last penalized team is behind by one goal or ahead by one goal. t or T: Time since last penalty is 2 or 7 minutes. n or N: Number of penalties is 3 or 12 (time in game is 10 or 55 minutes). h or H: Last penalty was not on the home team or it was. P: Parallel

42 biggest at grtnh: last penalized behind, just had two calls on them, has not been long since last call, early in the game, they are the home team. p(revcall) P: Parallel grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh grtnh p(revcall) Grtnh GrtnH GrtNh GrtNH GrTnh GrTnH GrTNh GrTNH GRtnh GRtnH GRtNh GRtNH GRTnh GRTnH GRTNh GRTNH Yes, Glen, referees are predictable!!! If you analyze these carefully, there are interesting interactions!!

43 P: Parallel Dave Higdon said, we tried your stuff (the R package BayesTree) on the analysis of computer experiments and it seemed promising but it is too slow. P: Parallel Recode with MPI to make it faster!! MPI: Message Passing Interface. Two Steps. Step 1. Rewrote serial code so that it is leaner.

44 Lean Code: class tree { public:... private: // //parameter for node double mu; // //rule: left if x[v] < xinfo[v][c] size_t v; size_t c; // //tree structure tree_p p; //parent tree_p l; //left child tree_p r; //right child }; P: Parallel 1 double, two integers, three pointers.

45 n bart/bayestree MCMC new MCMC 1, , , , , , , , , , P: Parallel The new code is 4 to 6 times faster!! > / [1] > / [1] Note: We also use a more limited set of MCMC moves in the new code but we find we get the same fits.

46 Step 2: Parallel MPI implementation. Have p + 1 processor cores. Split data up into p equal chunks. core 0 is the master. It runs the MCMC. P: Parallel core i, i = 1, 2,..., p has data chunk i in memory. Each core has the complete model ((T j, M j ) m j=1, σ) in memory. Note: with MPI cores and associated memory may be on different machines. Compare with openmp where the memory must be shared.

47 core 0 is the master. It runs the MCMC. core i, i = 1, 2,..., p has data chunk i in memory. Each core has the complete model ((T j, M j ) m j=1, σ) in memory. Each MCMC step involves: 1. master core 0, initiates an MCMC step (e.g. change a single tree, draw σ). P: Parallel 2. master core 0, sends out a compute request (needed for MCMC step) to each slave node i = 1, 2,... p. 3. Each slave core i computes on it s part of the data and sends the results back to master core master core 0 combines the results from the p slaves and updates the model using the results (e.g. changes a tree, obtains new σ draw). 5. master core 0, copies new model state out to all the slave cores.

48 Keys to Parallel implementation: Lean model representation is cheap to copy out to all the slave cores. MCMC draws all depend on simple conditionally sufficient statistics which may be computed on the separate slave cores, cheaply sent back to the master, and then combined. P: Parallel Even though the the overall model is complex, each local move is simple!!

49 Simulating p(t data) with the Bayesian CART Algorithm Simple Sufficient Statistics: Because p(t data) is available in closed form (up to a norming constant), we use a Metropolis-Hastings algorithm. Consider the birth-death step: Our proposal moves around tree space by proposing local modifications such as? =>? => propose a more complex tree propose a simpler tree P: Parallel Such modifications are accepted according to their compatibility with p(t data). Given a tree, we just have {R ij } N(µ j, σ 2 ) iid in j th 20 bottom node, where R is resids from the the other trees.. Evaluating a birth-death is just like testing equality of two normal means with an independent normal prior. Sufficient statistic is just i r ij for two different j corresponding to left/right child bottom nodes.

50 Timing We did lots of timing exercises to see how well it works. Here n = 200, 000, p + 1 = 40, time is to do 20,000 iterations. processors run time (s) P: Parallel 40/8=5. 39/7 = /4985=7.5. With 5 times as many processors we are 7.5 times faster.

51 Here is a plot of the times. T ser : time with serial code; T par : time with parallel code. Ideally you might hope for So we plot 1 T par vs. p + 1. T par = T ser p + 1. P: Parallel Looks pretty linear. Jump at about p + 1 = 30. 1/time number of processors

52 Here we look at the efficiency which is E = T ser (p + 1)T par If T par = Tser (p+1) this would equal 1. We timed 27 runs with m {50, 100, 200}, n {100000, , }, p + 1 {9, 17, 25}. P: Parallel If you have too few observations on a core the cost of message passing eats into the speed. 60% of p + 1 times faster is still pretty good! efficiency

53 At Google they were interested, but also politely scoffed. Their attitude is that on a typical large network the communication costs will really bite. Some of our timings were on a shared memory machine with 32 cores over which we had complete control, but some were on a cluster where you buy computes and these are over several machines. P: Parallel In any case, our setup may not work with the kind of machine setup they want to use at Google.

54 So, a simpler idea is consensus Bayes. Suppose you have a cluster of p nodes. Again, you split the data up into p chunks (no master this time!). Then you assign each node one of the chunks of data. Then you simply run a separate MCMC on each node using its chunk of data. P: Parallel If you want to predict at x, let f ij (x) be the i th draw of f on node j. Then you get a set of consensus draws by f c i (x) = w j f ij (x), w j = 1/Var(f ij (x)).

55 also involves an adjustment of the prior. The idea is that p(θ D) L(D)p(θ) Π p j=1 L(D j)p(θ) 1/p. where D is all the data and D j is the data on chunk j. P: Parallel So, you should use prior p(θ) 1/p on each node. This can work really well for simple parametric models, but is non-parametric with variable dimension.

56 Simulated data. n = 20,000. k = 10. p = 30. (1) True f (x) vs. serial ˆf (x). (2) True f (x) vs. consensus ˆf (x), no prior adjustment. (3) True f (x) vs. consensus ˆf (x), prior adjustment. P: Parallel bart mean consensus bart mean, no prior adjustment consensus bart mean, prior adjustment true f(x) true f(x) true f(x) Consensus without prior adjustment is awesome!

57 Top row: consensus no prior adjustment. Bottom row: consensus prior adjustment. First column, posterior mean of f (x) vs. serial posterior mean. Second column, 5% quantile vs serial. Third column, 95% quantile vs serial. consensus bart, no prior adjustment consensus bart, no prior adjustment consensus bart, no prior adjustment P: Parallel bart mean bart 5% bart 95% consensus bart, prior adjustment consensus bart, prior adjustment consensus bart, prior adjustment bart mean bart 5% bart 95% Consensus without prior adjustment is awesome!

58 Good: Both approaches make a usable technique with large data sets and seem to work very well. Need to think more about how to make prior adjustment (if any) for. P: Parallel Bad: While P is on my webpage, we need to make it easier to use. We have some R functions written and are testing, but it is not available yet.

### Data Mining with Bayesian Trees

Data Mining with Bayesian Rob McCulloch University of Chicago, Booth School of Business Milwaukee, April 4, 2014 P: Parallel Outline: (i) ensemble methods. (ii) : a Bayesian ensemble method. P: Parallel

### HT2015: SC4 Statistical Data Mining and Machine Learning

HT2015: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Bayesian Nonparametrics Parametric vs Nonparametric

### Bayesian Machine Learning (ML): Modeling And Inference in Big Data. Zhuhua Cai Google, Rice University caizhua@gmail.com

Bayesian Machine Learning (ML): Modeling And Inference in Big Data Zhuhua Cai Google Rice University caizhua@gmail.com 1 Syllabus Bayesian ML Concepts (Today) Bayesian ML on MapReduce (Next morning) Bayesian

### Supervised Learning (Big Data Analytics)

Supervised Learning (Big Data Analytics) Vibhav Gogate Department of Computer Science The University of Texas at Dallas Practical advice Goal of Big Data Analytics Uncover patterns in Data. Can be used

### Introduction to Machine Learning. Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011

Introduction to Machine Learning Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011 1 Outline 1. What is machine learning? 2. The basic of machine learning 3. Principles and effects of machine learning

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### Class #6: Non-linear classification. ML4Bio 2012 February 17 th, 2012 Quaid Morris

Class #6: Non-linear classification ML4Bio 2012 February 17 th, 2012 Quaid Morris 1 Module #: Title of Module 2 Review Overview Linear separability Non-linear classification Linear Support Vector Machines

### A Bootstrap Metropolis-Hastings Algorithm for Bayesian Analysis of Big Data

A Bootstrap Metropolis-Hastings Algorithm for Bayesian Analysis of Big Data Faming Liang University of Florida August 9, 2015 Abstract MCMC methods have proven to be a very powerful tool for analyzing

### Data Mining Practical Machine Learning Tools and Techniques

Ensemble learning Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 8 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Combining multiple models Bagging The basic idea

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### Knowledge Discovery and Data Mining. Bootstrap review. Bagging Important Concepts. Notes. Lecture 19 - Bagging. Tom Kelsey. Notes

Knowledge Discovery and Data Mining Lecture 19 - Bagging Tom Kelsey School of Computer Science University of St Andrews http://tom.host.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-19-B &

### COPYRIGHTED MATERIAL. Contents. List of Figures. Acknowledgments

Contents List of Figures Foreword Preface xxv xxiii xv Acknowledgments xxix Chapter 1 Fraud: Detection, Prevention, and Analytics! 1 Introduction 2 Fraud! 2 Fraud Detection and Prevention 10 Big Data for

### Homework Assignment 7

Homework Assignment 7 36-350, Data Mining Solutions 1. Base rates (10 points) (a) What fraction of the e-mails are actually spam? Answer: 39%. > sum(spam\$spam=="spam") [1] 1813 > 1813/nrow(spam) [1] 0.3940448

### Data Mining. Nonlinear Classification

Data Mining Unit # 6 Sajjad Haider Fall 2014 1 Nonlinear Classification Classes may not be separable by a linear boundary Suppose we randomly generate a data set as follows: X has range between 0 to 15

Statistics Graduate Courses STAT 7002--Topics in Statistics-Biological/Physical/Mathematics (cr.arr.).organized study of selected topics. Subjects and earnable credit may vary from semester to semester.

### Applied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets

Applied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets http://info.salford-systems.com/jsm-2015-ctw August 2015 Salford Systems Course Outline Demonstration of two classification

### BOOSTED REGRESSION TREES: A MODERN WAY TO ENHANCE ACTUARIAL MODELLING

BOOSTED REGRESSION TREES: A MODERN WAY TO ENHANCE ACTUARIAL MODELLING Xavier Conort xavier.conort@gear-analytics.com Session Number: TBR14 Insurance has always been a data business The industry has successfully

### Generalizing Random Forests Principles to other Methods: Random MultiNomial Logit, Random Naive Bayes, Anita Prinzie & Dirk Van den Poel

Generalizing Random Forests Principles to other Methods: Random MultiNomial Logit, Random Naive Bayes, Anita Prinzie & Dirk Van den Poel Copyright 2008 All rights reserved. Random Forests Forest of decision

### Classification and Regression Trees

Classification and Regression Trees Bob Stine Dept of Statistics, School University of Pennsylvania Trees Familiar metaphor Biology Decision tree Medical diagnosis Org chart Properties Recursive, partitioning

### IFT3395/6390. Machine Learning from linear regression to Neural Networks. Machine Learning. Training Set. t (3.5, -2,..., 127, 0,...

IFT3395/6390 Historical perspective: back to 1957 (Prof. Pascal Vincent) (Rosenblatt, Perceptron ) Machine Learning from linear regression to Neural Networks Computer Science Artificial Intelligence Symbolic

### Using multiple models: Bagging, Boosting, Ensembles, Forests

Using multiple models: Bagging, Boosting, Ensembles, Forests Bagging Combining predictions from multiple models Different models obtained from bootstrap samples of training data Average predictions or

### TRANSACTIONAL DATA MINING AT LLOYDS BANKING GROUP

TRANSACTIONAL DATA MINING AT LLOYDS BANKING GROUP Csaba Főző csaba.fozo@lloydsbanking.com 15 October 2015 CONTENTS Introduction 04 Random Forest Methodology 06 Transactional Data Mining Project 17 Conclusions

Neural Network Add-in Version 1.5 Software User s Guide Contents Overview... 2 Getting Started... 2 Working with Datasets... 2 Open a Dataset... 3 Save a Dataset... 3 Data Pre-processing... 3 Lagging...

### Government of Russian Federation. Faculty of Computer Science School of Data Analysis and Artificial Intelligence

Government of Russian Federation Federal State Autonomous Educational Institution of High Professional Education National Research University «Higher School of Economics» Faculty of Computer Science School

### Introduction to Machine Learning Lecture 1. Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu

Introduction to Machine Learning Lecture 1 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Introduction Logistics Prerequisites: basics concepts needed in probability and statistics

### Fast Analytics on Big Data with H20

Fast Analytics on Big Data with H20 0xdata.com, h2o.ai Tomas Nykodym, Petr Maj Team About H2O and 0xdata H2O is a platform for distributed in memory predictive analytics and machine learning Pure Java,

### Microsoft Azure Machine learning Algorithms

Microsoft Azure Machine learning Algorithms Tomaž KAŠTRUN @tomaz_tsql Tomaz.kastrun@gmail.com http://tomaztsql.wordpress.com Our Sponsors Speaker info https://tomaztsql.wordpress.com Agenda Focus on explanation

### Big Data, Statistics, and the Internet

Big Data, Statistics, and the Internet Steven L. Scott April, 4 Steve Scott (Google) Big Data, Statistics, and the Internet April, 4 / 39 Summary Big data live on more than one machine. Computing takes

### Why Ensembles Win Data Mining Competitions

Why Ensembles Win Data Mining Competitions A Predictive Analytics Center of Excellence (PACE) Tech Talk November 14, 2012 Dean Abbott Abbott Analytics, Inc. Blog: http://abbottanalytics.blogspot.com URL:

### Machine Learning and Data Analysis overview. Department of Cybernetics, Czech Technical University in Prague. http://ida.felk.cvut.

Machine Learning and Data Analysis overview Jiří Kléma Department of Cybernetics, Czech Technical University in Prague http://ida.felk.cvut.cz psyllabus Lecture Lecturer Content 1. J. Kléma Introduction,

### C19 Machine Learning

C9 Machine Learning 8 Lectures Hilary Term 25 2 Tutorial Sheets A. Zisserman Overview: Supervised classification perceptron, support vector machine, loss functions, kernels, random forests, neural networks

### Neural Networks & Boosting

Neural Networks & Boosting Bob Stine Dept of Statistics, School University of Pennsylvania Questions How is logistic regression different from OLS? Logistic mean function for probabilities Larger weight

### Classification of Bad Accounts in Credit Card Industry

Classification of Bad Accounts in Credit Card Industry Chengwei Yuan December 12, 2014 Introduction Risk management is critical for a credit card company to survive in such competing industry. In addition

### Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05

Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification

### Data Mining - Evaluation of Classifiers

Data Mining - Evaluation of Classifiers Lecturer: JERZY STEFANOWSKI Institute of Computing Sciences Poznan University of Technology Poznan, Poland Lecture 4 SE Master Course 2008/2009 revised for 2010

### Leveraging Ensemble Models in SAS Enterprise Miner

ABSTRACT Paper SAS133-2014 Leveraging Ensemble Models in SAS Enterprise Miner Miguel Maldonado, Jared Dean, Wendy Czika, and Susan Haller SAS Institute Inc. Ensemble models combine two or more models to

### Some Examples of (Markov Chain) Monte Carlo Methods

Some Examples of (Markov Chain) Monte Carlo Methods Ryan R. Rosario What is a Monte Carlo method? Monte Carlo methods rely on repeated sampling to get some computational result. Monte Carlo methods originated

### An Introduction to Data Mining

An Introduction to Intel Beijing wei.heng@intel.com January 17, 2014 Outline 1 DW Overview What is Notable Application of Conference, Software and Applications Major Process in 2 Major Tasks in Detail

### MERGING BUSINESS KPIs WITH PREDICTIVE MODEL KPIs FOR BINARY CLASSIFICATION MODEL SELECTION

MERGING BUSINESS KPIs WITH PREDICTIVE MODEL KPIs FOR BINARY CLASSIFICATION MODEL SELECTION Matthew A. Lanham & Ralph D. Badinelli Virginia Polytechnic Institute and State University Department of Business

### Machine Learning Methods for Demand Estimation

Machine Learning Methods for Demand Estimation By Patrick Bajari, Denis Nekipelov, Stephen P. Ryan, and Miaoyu Yang Over the past decade, there has been a high level of interest in modeling consumer behavior

Graduate Programs in Statistics Course Titles STAT 100 CALCULUS AND MATR IX ALGEBRA FOR STATISTICS. Differential and integral calculus; infinite series; matrix algebra STAT 195 INTRODUCTION TO MATHEMATICAL

### Principles of Data Mining by Hand&Mannila&Smyth

Principles of Data Mining by Hand&Mannila&Smyth Slides for Textbook Ari Visa,, Institute of Signal Processing Tampere University of Technology October 4, 2010 Data Mining: Concepts and Techniques 1 Differences

### Learning outcomes. Knowledge and understanding. Competence and skills

Syllabus Master s Programme in Statistics and Data Mining 120 ECTS Credits Aim The rapid growth of databases provides scientists and business people with vast new resources. This programme meets the challenges

### Scalable Developments for Big Data Analytics in Remote Sensing

Scalable Developments for Big Data Analytics in Remote Sensing Federated Systems and Data Division Research Group High Productivity Data Processing Dr.-Ing. Morris Riedel et al. Research Group Leader,

### Data Mining: Overview. What is Data Mining?

Data Mining: Overview What is Data Mining? Recently * coined term for confluence of ideas from statistics and computer science (machine learning and database methods) applied to large databases in science,

### Neural Networks for Machine Learning. Lecture 13a The ups and downs of backpropagation

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed A brief history of backpropagation

### Fortgeschrittene Computerintensive Methoden

Fortgeschrittene Computerintensive Methoden Einheit 3: mlr - Machine Learning in R Bernd Bischl Matthias Schmid, Manuel Eugster, Bettina Grün, Friedrich Leisch Institut für Statistik LMU München SoSe 2014

### BayesX - Software for Bayesian Inference in Structured Additive Regression

BayesX - Software for Bayesian Inference in Structured Additive Regression Thomas Kneib Faculty of Mathematics and Economics, University of Ulm Department of Statistics, Ludwig-Maximilians-University Munich

### The Operational Value of Social Media Information. Social Media and Customer Interaction

The Operational Value of Social Media Information Dennis J. Zhang (Kellogg School of Management) Ruomeng Cui (Kelley School of Business) Santiago Gallino (Tuck School of Business) Antonio Moreno-Garcia

### Learning is a very general term denoting the way in which agents:

What is learning? Learning is a very general term denoting the way in which agents: Acquire and organize knowledge (by building, modifying and organizing internal representations of some external reality);

### MS1b Statistical Data Mining

MS1b Statistical Data Mining Yee Whye Teh Department of Statistics Oxford http://www.stats.ox.ac.uk/~teh/datamining.html Outline Administrivia and Introduction Course Structure Syllabus Introduction to

### Comparison of Data Mining Techniques used for Financial Data Analysis

Comparison of Data Mining Techniques used for Financial Data Analysis Abhijit A. Sawant 1, P. M. Chawan 2 1 Student, 2 Associate Professor, Department of Computer Technology, VJTI, Mumbai, INDIA Abstract

### Car Insurance. Prvák, Tomi, Havri

Car Insurance Prvák, Tomi, Havri Sumo report - expectations Sumo report - reality Bc. Jan Tomášek Deeper look into data set Column approach Reminder What the hell is this competition about??? Attributes

### Learning. Artificial Intelligence. Learning. Types of Learning. Inductive Learning Method. Inductive Learning. Learning.

Learning Learning is essential for unknown environments, i.e., when designer lacks omniscience Artificial Intelligence Learning Chapter 8 Learning is useful as a system construction method, i.e., expose

### Knowledge Discovery and Data Mining

Knowledge Discovery and Data Mining Unit # 11 Sajjad Haider Fall 2013 1 Supervised Learning Process Data Collection/Preparation Data Cleaning Discretization Supervised/Unuspervised Identification of right

### CI6227: Data Mining. Lesson 11b: Ensemble Learning. Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore.

CI6227: Data Mining Lesson 11b: Ensemble Learning Sinno Jialin PAN Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore Acknowledgements: slides are adapted from the lecture notes

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### REVIEW OF ENSEMBLE CLASSIFICATION

Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology ISSN 2320 088X IJCSMC, Vol. 2, Issue.

### Model Combination. 24 Novembre 2009

Model Combination 24 Novembre 2009 Datamining 1 2009-2010 Plan 1 Principles of model combination 2 Resampling methods Bagging Random Forests Boosting 3 Hybrid methods Stacking Generic algorithm for mulistrategy

10-601 Machine Learning http://www.cs.cmu.edu/afs/cs/academic/class/10601-f10/index.html Course data All up-to-date info is on the course web page: http://www.cs.cmu.edu/afs/cs/academic/class/10601-f10/index.html

### An Overview of Knowledge Discovery Database and Data mining Techniques

An Overview of Knowledge Discovery Database and Data mining Techniques Priyadharsini.C 1, Dr. Antony Selvadoss Thanamani 2 M.Phil, Department of Computer Science, NGM College, Pollachi, Coimbatore, Tamilnadu,

### New Work Item for ISO 3534-5 Predictive Analytics (Initial Notes and Thoughts) Introduction

Introduction New Work Item for ISO 3534-5 Predictive Analytics (Initial Notes and Thoughts) Predictive analytics encompasses the body of statistical knowledge supporting the analysis of massive data sets.

### The Artificial Prediction Market

The Artificial Prediction Market Adrian Barbu Department of Statistics Florida State University Joint work with Nathan Lay, Siemens Corporate Research 1 Overview Main Contributions A mathematical theory

### Insurance Analytics - analýza dat a prediktivní modelování v pojišťovnictví. Pavel Kříž. Seminář z aktuárských věd MFF 4.

Insurance Analytics - analýza dat a prediktivní modelování v pojišťovnictví Pavel Kříž Seminář z aktuárských věd MFF 4. dubna 2014 Summary 1. Application areas of Insurance Analytics 2. Insurance Analytics

### CART 6.0 Feature Matrix

CART 6.0 Feature Matri Enhanced Descriptive Statistics Full summary statistics Brief summary statistics Stratified summary statistics Charts and histograms Improved User Interface New setup activity window

### Data Mining with R. December 13, 2008

Data Mining with R John Maindonald (Centre for Mathematics and Its Applications, Australian National University) and Yihui Xie (School of Statistics, Renmin University of China) December 13, 2008 Data

### Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones

Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 What is machine learning? Data description and interpretation

### Statistical machine learning, high dimension and big data

Statistical machine learning, high dimension and big data S. Gaïffas 1 14 mars 2014 1 CMAP - Ecole Polytechnique Agenda for today Divide and Conquer principle for collaborative filtering Graphical modelling,

### Outlier Ensembles. Charu C. Aggarwal IBM T J Watson Research Center Yorktown, NY 10598. Keynote, Outlier Detection and Description Workshop, 2013

Charu C. Aggarwal IBM T J Watson Research Center Yorktown, NY 10598 Outlier Ensembles Keynote, Outlier Detection and Description Workshop, 2013 Based on the ACM SIGKDD Explorations Position Paper: Outlier

### A Study Of Bagging And Boosting Approaches To Develop Meta-Classifier

A Study Of Bagging And Boosting Approaches To Develop Meta-Classifier G.T. Prasanna Kumari Associate Professor, Dept of Computer Science and Engineering, Gokula Krishna College of Engg, Sullurpet-524121,

### Gerry Hobbs, Department of Statistics, West Virginia University

Decision Trees as a Predictive Modeling Method Gerry Hobbs, Department of Statistics, West Virginia University Abstract Predictive modeling has become an important area of interest in tasks such as credit

### Predictive Modeling with The R CARET Package

Predictive Modeling with The R CARET Package Matthew A. Lanham, CAP (lanham@vt.edu) Doctoral Candidate Department of Business Information Technology MatthewALanham.com R is a software environment for data

### Better credit models benefit us all

Better credit models benefit us all Agenda Credit Scoring - Overview Random Forest - Overview Random Forest outperform logistic regression for credit scoring out of the box Interaction term hypothesis

### Lecture 6. Artificial Neural Networks

Lecture 6 Artificial Neural Networks 1 1 Artificial Neural Networks In this note we provide an overview of the key concepts that have led to the emergence of Artificial Neural Networks as a major paradigm

### Chapter 6. The stacking ensemble approach

82 This chapter proposes the stacking ensemble approach for combining different data mining classifiers to get better performance. Other combination techniques like voting, bagging etc are also described

### Bayesian networks - Time-series models - Apache Spark & Scala

Bayesian networks - Time-series models - Apache Spark & Scala Dr John Sandiford, CTO Bayes Server Data Science London Meetup - November 2014 1 Contents Introduction Bayesian networks Latent variables Anomaly

### Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

### Beating the MLB Moneyline

Beating the MLB Moneyline Leland Chen llxchen@stanford.edu Andrew He andu@stanford.edu 1 Abstract Sports forecasting is a challenging task that has similarities to stock market prediction, requiring time-series

### Sampling via Moment Sharing: A New Framework for Distributed Bayesian Inference for Big Data

Sampling via Moment Sharing: A New Framework for Distributed Bayesian Inference for Big Data (Oxford) in collaboration with: Minjie Xu, Jun Zhu, Bo Zhang (Tsinghua) Balaji Lakshminarayanan (Gatsby) Bayesian

### NEURAL NETWORKS A Comprehensive Foundation

NEURAL NETWORKS A Comprehensive Foundation Second Edition Simon Haykin McMaster University Hamilton, Ontario, Canada Prentice Hall Prentice Hall Upper Saddle River; New Jersey 07458 Preface xii Acknowledgments

### Machine Learning Big Data using Map Reduce

Machine Learning Big Data using Map Reduce By Michael Bowles, PhD Where Does Big Data Come From? -Web data (web logs, click histories) -e-commerce applications (purchase histories) -Retail purchase histories

### Bayesian Statistics: Indian Buffet Process

Bayesian Statistics: Indian Buffet Process Ilker Yildirim Department of Brain and Cognitive Sciences University of Rochester Rochester, NY 14627 August 2012 Reference: Most of the material in this note

### Predicting required bandwidth for educational institutes using prediction techniques in data mining (Case Study: Qom Payame Noor University)

260 IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.6, June 2011 Predicting required bandwidth for educational institutes using prediction techniques in data mining (Case

### Learning outcomes. Knowledge and understanding. Ability and Competences. Evaluation capability and scientific approach

Syllabus Master s Programme in Statistics and Data Mining 120 ECTS Credits Aim The rapid growth of databases provides scientists and business people with vast new resources. This programme meets the challenges

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Making Sense of the Mayhem: Machine Learning and March Madness

Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research

### Classification and Regression by randomforest

Vol. 2/3, December 02 18 Classification and Regression by randomforest Andy Liaw and Matthew Wiener Introduction Recently there has been a lot of interest in ensemble learning methods that generate many

### Parallelization Strategies for Multicore Data Analysis

Parallelization Strategies for Multicore Data Analysis Wei-Chen Chen 1 Russell Zaretzki 2 1 University of Tennessee, Dept of EEB 2 University of Tennessee, Dept. Statistics, Operations, and Management

### Solving Regression Problems Using Competitive Ensemble Models

Solving Regression Problems Using Competitive Ensemble Models Yakov Frayman, Bernard F. Rolfe, and Geoffrey I. Webb School of Information Technology Deakin University Geelong, VIC, Australia {yfraym,brolfe,webb}@deakin.edu.au

### Active Learning SVM for Blogs recommendation

Active Learning SVM for Blogs recommendation Xin Guan Computer Science, George Mason University Ⅰ.Introduction In the DH Now website, they try to review a big amount of blogs and articles and find the

### Decompose Error Rate into components, some of which can be measured on unlabeled data

Bias-Variance Theory Decompose Error Rate into components, some of which can be measured on unlabeled data Bias-Variance Decomposition for Regression Bias-Variance Decomposition for Classification Bias-Variance

### Neural Networks. Neural network is a network or circuit of neurons. Neurons can be. Biological neurons Artificial neurons

Neural Networks Neural network is a network or circuit of neurons Neurons can be Biological neurons Artificial neurons Biological neurons Building block of the brain Human brain contains over 10 billion

### IST 557 Final Project

George Slota DataMaster 5000 IST 557 Final Project Abstract As part of a competition hosted by the website Kaggle, a statistical model was developed for prediction of United States Census 2010 mailing

### 6.2.8 Neural networks for data mining

6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural

### Machine Learning and Data Mining. Fundamentals, robotics, recognition

Machine Learning and Data Mining Fundamentals, robotics, recognition Machine Learning, Data Mining, Knowledge Discovery in Data Bases Their mutual relations Data Mining, Knowledge Discovery in Databases,

### Predictive Modeling Techniques in Insurance

Predictive Modeling Techniques in Insurance Tuesday May 5, 2015 JF. Breton Application Engineer 2014 The MathWorks, Inc. 1 Opening Presenter: JF. Breton: 13 years of experience in predictive analytics

### Chapter 11 Boosting. Xiaogang Su Department of Statistics University of Central Florida - 1 -

Chapter 11 Boosting Xiaogang Su Department of Statistics University of Central Florida - 1 - Perturb and Combine (P&C) Methods have been devised to take advantage of the instability of trees to create

### One Statistician s Perspectives on Statistics and "Big Data" Analytics

One Statistician s Perspectives on Statistics and "Big Data" Analytics Some (Ultimately Unsurprising) Lessons Learned Prof. Stephen Vardeman IMSE & Statistics Departments Iowa State University July 2014