Exploratory Data Analysis

Transcription

1 Exploratory Data Analysis Paul Cohen ISTA 370 Spring, 2012 Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

2 Outline Data, revisited The purpose of exploratory data analysis Learning to see Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

3 Data: A Review Things and Data Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

4 Data: A Review Things and Data Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

5 Data: A Review Where Data Come From Data are measurements of individuals (people, trees, countries, ecosystems...). An Individual Data 56 years old 70" tall 180 lbs Brown eyes Moderately presbyo8c Good health Married One child... A Data Table Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

6 Exploratory Data Analysis What is Exploratory Data Analysis (EDA)? In terms of the Fundamental Model of Data, y = f (x, ɛ): EDA infers which factors strongly and weakly influence y and the functions that combine these factors EDA examines ɛ to see whether it contains evidence of other important but unmeasured ( hidden ) factors Confirmatory studies test whether x really is a causal factor that influences y Exploratory studies are to confirmatory studies as test kitchens are to cookbook recipes. EDA generally doesn t test hypotheses, but, rather, helps the data tell its story EDA helps you understand phenomena, and suggests hypotheses to test in confirmatory studies. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

7 Exploratory Data Analysis What is Exploratory Data Analysis? Learning to See Data have structure that is evidence of causal influences. EDA uncovers, exposes, clarifies this structure. EDA is like hunting for fossils it s a skill, and you must learn to see not only what s in front of you, but what lies within data. EDA asks, what do I see, and what does it mean? Like any other skill, EDA takes a lot of practice. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

8 Exploratory Data Analysis Load Some Data > read.table.ista370<-function(filename){ dataurl<-" # Reads a data frame from a URL path rooted at ISTA370 dat read.table(paste(dataurl,filename,sep="")) } > > # taheri<-read.table.ista370("taheri1.txt") > # iris<-read.table.ista370("iris.txt") > # heightc<-read.table.ista370("heightc.txt") > # treering<-read.table.ista370("treering1.txt") > # blast<-read.table.ista370("blastsummary.txt") > # kinect<-read.table.ista370("onemovie.txt") > # readability<-read.table.ista370("readability.txt") Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

9 What Do You See? What Does it Mean? > hist(iris$petal.length,col="grey",main=null) Frequency iris$petal.length Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

10 What Do You See? What Does it Mean? > ipl<-iris$petal.length > hist(ipl,prob="true",ylim=c(0,1),main=null) > lines(density(ipl[iris$species=="setosa"]),col="red") > lines(density(ipl[iris$species=="versicolor"]),col="green") > lines(density(ipl[iris$species=="virginica"]),col="blue") Density Looking at density curves for each species, we see that the histogram did indeed indicate two or more separate populations (species) ipl Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

11 What Do You See? What Does it Mean? > boxplot(iris$petal.length~iris$species, ylab="petal.length",xlab="species") Petal.Length A boxplot by species confirms, and summarizes the petal length statistics for each species. setosa versicolor virginica Species Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

12 Boxplots outliers whisker (various interpreta1ons) upper quar,le (75% quan,le) median interquar,le range lower quar,le (25% quan,le) whisker outliers Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

13 Median, Quartiles, Interquartile Range If you sort the values in a sample from lowest to highest, the median is the middle value, or the average of the two middle values when the sample contains an even number of points. The median is the 50th quantile, or the value for which 50% of the values are greater. The lower quartile is the 25th quantile, above which 75% of the values are found. The upper quartile is the 75th quantile, above which 25% of the values are found. The interquartile range is a measure of variability and is the difference between the upper and lower quartiles. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

14 Median, Quartiles, Interquartile Range The median is robust against outliers; the mean is not. Suppose 100 families in a neighborhood each make $40,000/year. When a millionaire moves in the mean jumps from $40,000 to $49,504/year. What happens to the median? Before the millionaire arrived, the variance in income was zero. Afterwards the variance is over nine billion!!! What happens to the interquartile range? Suppose you have a sorted sample of 9 elements; the median is the fifth element. If you add another element, what will the median be? By how many locations in the sorted distribution can the median shift? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

15 Symmetry and Skew > with(blast, hist(test0,breaks=20,col="grey",main=null)) > with(treering, hist(width,breaks=20,col="grey",main=null)) Frequency Frequency Test width Test0 is skewed to the right, meaning it has a long tail of higher values, while Treering is nearly symmetric. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

16 Transformations > attach(blast) > hist(train0,breaks=20,col="grey",main=null) > Train0Squared<-Train0^2 #square the Train0 data > hist(train0squared,breaks=20,col="grey",main=null) Frequency Frequency Train Train0Squared A simple transformation amplifies an otherwise hidden feature Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

17 Transformations > Train0Squared<-with(blast,Train0^2) > with(blast,plot(density(train0squared))) > with(blast,lines(density(train0squared[gender=="female"]),c > with(blast,lines(density(train0squared[gender=="male"]),col density.default(x = Train0Squared) Density N = 187 Bandwidth = Does gender explain the bump? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

18 What Explains the Bump New Topics > plot(train0squared,newskills0,col=gender) > plot(newskills0,train0squared,col=gender) Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46 The number of topics to which students were exposed (NewSkills0) seems to explain the bump, but gender doesn t NewSkills0 Train0Squared Train0Squared NewSkills0

19 What Explains the Bump New Topics > precocious<-newskills0>8 > plot(density(train0squared)) > lines(density(train0squared[precocious=="true"]),col="red") > lines(density(train0squared[precocious=="false"]),col="blue density.default(x = Train0Squared) Density N = 187 Bandwidth = So the students who saw too many subjects account for the bump. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

20 Boxplots instead of density plots > boxplot(train0squared~precocious, xlab="precocious", ylab="proportion training items correct" ) proportion training items correct FALSE TRUE precocious Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

21 What Explains the Bump Why did some students see hard problems? Exploratory data analysis helped us find and amplify an odd feature of data: Some students saw too many hard problems while training for the first test. How did this happen? > table(precocious, policy) policy precocious DBN_11 EXPERT RANDOM FALSE TRUE All these precocious students were in one condition of the experiment. In the RANDOM condition, training problems were selected at random, so we shouldn t be surprised that students in this condition bombed on the first test! Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

22 Data, revisited Outline The purpose of exploratory data analysis Learning to see: Histograms, boxplots, median and other robust statistics, transforming data...missing values, tips. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

23 What Do You See? What Does it Mean? > ts.plot(kinect$lhand.y,ylim=c(-500,1000),col="red") kinect$lhand.y Time Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

24 What Do You See? What Does it Mean? > ts.plot(kinect$lhand.y,ylim=c(-500,1000),col="red") > lines(rep(0,150)) kinect$lhand.y Constants and lack of change are rare in biometric data. Zero is a special number. Perhaps the Kinect codes missing data as 0. What is really happening around time 115? Time Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

25 Missing Data Data can be missing for many reasons (e.g., subjects in BLAST experiment were allowed to leave once they hit a maximum score, so didn t take all tests) Missing data is usually given a code, such as -999 or NA. The Kinect code of zero is unhelpful. Why? R sometimes refuses to do math on random variables with missing values. Is this unhelpful? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

26 Missing Data in R > # won't work, missing values: > with(blast,mean(test3)) [1] NA > # this time, exclude missing values: > with(blast,mean(test3,na.rm=true)) [1] > # get their indices: > with(blast,which(is.na(test3))) [1] Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

27 Missing Data: It Matters Why! Most experiment results are based on the assumption of random sampling or random assignment to conditions. When data are Missing Completely At Random (MCAR), missing data don t violate these assumptions. MCAR means missingness is independent of measured and unmeasured factors. Data are Missing at Random (MAR) when the reason they are missing has nothing to do with the data themselves. If food poisoning is proportional to fast food consumption (FFC), but FFC is unrelated to enrollment in ISTA 370, then if you re absent due to food poisoning, then you are MAR. NMAR data are not missing at random. If students ordinarily take four tests, but are excused from future tests if they get a perfect score on a test, are they MAR or NMAR? How might NMAR introduce errors in analysis? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

28 Not Missing At Random (NMAR) Data Participants in the BLAST experiment took four tests, but those who scored 18 or more on any test were excused from later tests. > which(is.na(test3)) # who didn't take Test3 [1] > Test2[which(is.na(Test3))] # what were their Test2 scores [1] 1.00 NA 0.95 NA > T3<-Test3 > mean(t3,na.rm=true) # Mean Test3 scores [1] > T3[is.na(T3)]<-1 # Replace NAs with perfect scores > mean(t3) # Mean T3 scores [1] Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

29 Not Missing At Random (NMAR) Data Does it Matter? If NMAR data are evenly distributed over experimental conditions, then they might not matter so much. So let s check: > nottest3<-which(is.na(test3)) # who didn't take Test3 > gender[nottest3] [1] female male male male female male male male Levels: female male > cond.plus.policy[nottest3] [1] DBN_11_NO_CHOICE DBN_11_NO_CHOICE DBN_11_NO_CHOICE [4] EXPERT_NO_CHOICE EXPERT_NO_CHOICE EXPERT_NO_CHOICE [7] EXPERT_CHOICE EXPERT_CHOICE_ZPD 6 Levels: DBN_11_NO_CHOICE... RANDOM_NO_CHOICE Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

30 Not Missing At Random (NMAR) Data Does it Matter? > aggregate.data.frame(test3,by=list(condition,gender), FUN=mean, na.rm=true) Group.1 Group.2 x 1 CHOICE female NO_CHOICE female CHOICE male NO_CHOICE male > aggregate.data.frame(t3,by=list(condition,gender), FUN=mean, na.rm=true) Group.1 Group.2 x 1 CHOICE female NO_CHOICE female CHOICE male NO_CHOICE male Note useful aggregate.data.frame command, which applies FUN to subsets of a variable defined by by=... T3 sets all the NAs to 1.0 so it s what students would get if they maxed out tests they were allowed to skip. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

31 Missing and Censored Data When a small fraction of your data is missing, you can ignore it or impute its values. Common imputation methods involve matching records that have missing data to records that don t, and using one or more of the complete records to infer the missing value. Censored data is a more challenging problem. Examples: Inferring average runtime of an algorithm for a batch of jobs that s allowed to run for a fixed time. Inferring age at death of a treatment sample when some people are still alive when the study ends. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

32 What Do You See? What Does it Mean? > hist(heightc$weight,main=null,xlab="weight of 33 College St Frequency Weight of 33 College Students Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

33 What Do You See? What Does it Mean? Frequency Common sense tells us that a weight of 50lbs or less is unlikely and is probably an errorful datum Weight of 33 College Students Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

34 What Do You See? What Does it Mean? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

35 What Do You See? What Does it Mean? The vertical axes are different, so it s hard to tell what s happening. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

36 What Do You See? What Does it Mean? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

37 What Do You See? What Does it Mean? Not all apparent differences are real differences. Mentally group your data to see if something might explain apparent differences. Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

38 What Do You See? What Does it Mean? > m<-c(0.93,0.95,0.94,0.86,na,0.86, 0.89, 0.85, 0.90, 0.85, 0 > s<-c(0.26, 0.23, 0.25, 0.34,NA,0.34, 0.31, 0.35, 0.30, 0.35 > barx <- barplot(m,ylim=c(0,1.5),names.arg=1:11,axis.lty=1,x > error.bar(barx,m,s) Not all differences are real differences Condition Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

39 What Do You See? What Does it Mean? > plot(taheri$a,col="red",type="l",ylim=c(0,1)) > lines(taheri$b,col="blue") taheri$a Index Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

40 What Do You See? What Does it Mean? > plot(taheri$a,col="red",type="l",ylim=c(0,1)) > lines(taheri$b,col="blue") > cor(taheri$a,taheri$b) [1] taheri$a Although features A and B were supposed to be independent, they were normalized to sum to a constant, rendering them dependent Index Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

41 What Do You See? What Does it Mean? > with(mtcars,plot(disp,mpg)) > cor(disp,mpg) [1] mpg disp Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

42 What Do You See? What Does it Mean? > palette(c("blue","red","forestgreen")) > with(mtcars,plot(disp,mpg,col=cyl)) > cor(disp,mpg) [1] mpg Coloring by a third variable shows that the correlation between disp and mpg depends on cyl disp Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

43 Tips for looking at data Tips: Not All Data are Real Data Data can be noisy, missing, contaminated, or even intentionally wrong (e.g., perverse subjects). You rarely know which data are suspicious. You have to look carefully for strange values and decide what to do with them. kinect$lhand.y Frequency Time Weight of 33 College Students Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

44 Tips for looking at data Tips: Not All Differences are Real Differences Get the right vertical axis scale Augment your picture with measures of variation Condition Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46

45 Tips for looking at data Tips: Look for holes in data Holes are regions that have fewer data. You ask, why are so few data there? Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / Train0Squared NewSkills0 iris$petal.length Frequency

46 Tips for looking at data Tips: Don t rely on summaries. Look at the data! Means and other summaries are useful but the raw data show patterns obscured by summaries. unemployment year Paul Cohen ISTA 370 () Exploratory Data Analysis Spring, / 46