Applied data mining" data exploration & regression

Size: px

Start display at page:

Download "Applied data mining" data exploration & regression"

Alexandra Patterson
7 years ago
Views:

1 Applied data mining" data exploration & regression Byron C Wallace Some content from

2 Data exploration Before any analysis, you should usually start by calculating simple summary stats and visualization your data I ll show some fun visualizations now

3 flowingdata.com 9/9/11

4 NYTimes 7/26/11

5 Exploratory Data Analysis (EDA) Get a general sense of the data means, medians, quintiles, histograms, boxplots " Data-driven (model-free) " Interactive and visual Humans are the best pattern recognizers Use more than 2 dimensions! x,y,z, space, color, time " Especially useful in early stages of data mining detect outliers (e.g. assess data quality) test assumptions (e.g. normal distributions or skewed?) identify useful raw data & transforms (e.g. log(x)) Always look at your data

6 Single Variable Visualization Histogram: shows center, variability, skewness, modality, outliers, or strange patterns. bins matter

7 But be careful with axes and scales!

8 Smoothed Histograms: Density plots

9 KDEs sum and normalize

10 To notebook!

11 Boxplots By convention, top and bottom of box are 1 st and 3 rd quartiles Shows a lot of information about a variable in one plot Median IQR Outliers Range Skewness" Negatives Overplotting Hard to tell distributional shape no standard implementation in software (many options for whiskers, outliers)

15 Time Series If your data has a temporal component, be sure to exploit it summer peaks steady growth trend New Year bumps Data Mining Volinsky - Columbia University

16 Spatial Data If your data has a geographic component, be sure to exploit it Data from cities/states/zip cods easy to get lat/ long Can plot as scatterplot

17 OK, your turn. 1. Download the plotting-and-viz.ipynb notebook from Canvas (under in-class-exercises/plottingand-viz.ipynb) or from GitHub. 2. Work through this (execute each line) make sure you understand what s going on! EXPERIMENT and change things, etc.! " - If anything is unclear or you just have questions about anything -- wave me down." 3. Also complete the interspersed exercise.

18 Supervised learning labeled data L learner predictive model unlabeled data U

19 Classification v Regression Classification Predict discrete labels (spam / not spam) Regression Predict continuous outcomes (e.g., house price)

20 Regression

21 Fitting lines

22 The aim We would like to find a line that minimizes the residuals (errors) Usually in terms of the sum of squared errors This is called the line of best fit and it minimizes the sum of squared residuals

23 How to pick a line, given points?

25 Select to minimize " sum of squared errors SSE = N ( y ) i b0 b1 x i i= 1 2

26 Regression

27 Grus says 0 = y -- x = corr(x, y) * stddev (y) / stddev (x) but where did these come from?

28 Other estimation methods So we saw (almost) how to solve this analytically but not always so easy Iterative methods, such as gradient descent, are widely used instead Simple, easy to implement, work well

29 The basic idea: to find a (local) function minimum, move in the negative direction of the gradient Gradient descent

30 The meta-algorithm do until convergence Loss function θ j := θ j α θ j J(θ). parameters step size

31 In Python

32 Assumptions Relationship between y and x is linear The errors do not vary with x Residuals are normally distributed

33 Hypothesis Testing and Regression We are often interested in establishing the relationship between variables Formally we can do this within the regression framework

34 Does x correlate with y? We can specify a model: y i = b 0 + b 1 x i Then we ask: is b 1 = 0 (this becomes H 0 )

35 For example Coefficients: (Intercept) housing$lotsize housing$lotsize 6.599e e <2e-16 ***

36 2 R

37 A word of caution on coefficients

38 Anscombe's quartet

40 Multiple regression Usually we have more than one predictor variable (jargon alert! we call these features) y i = b 0 + bx i Assumptions: linear relationship; multivariate normality; no or little colllinearity

41 Example in R" (bathrooms matter!)

42 Lab time! (Class exercise)

43 Categorical features:" dummy coding How we put categorical features into (multiple) regression models? Example: suppose we want to include sex as a predictor ( male / female ) how do we put this in the model? We insert an indicator variable that is 1 if the person represented by a row is female and 0 otherwise

44 Dealing with non-linear features Suppose we are predicting the number of days a house will remain on the market (on board) Consider also predicting an overall health or fitness score

45 Interaction features We may believe that some features interact with others E.g., maybe we want a feature crossing price bin indicator with a recently renovated or >= 3br indicator

Exercise 1.12 (Pg. 22-23)

Exercise 1.12 (Pg. 22-23) Individuals: The objects that are described by a set of data. They may be people, animals, things, etc. (Also referred to as Cases or Records) Variables: The characteristics recorded about each individual.