An Introduction to Statistical Learning

Transcription

1 Springer Texts in Statistics Gareth James Daniela Witten Trevr Hastie Rbert Tibshirani An Intrductin t Statistical Learning with Applicatins in R

2 Springer Texts in Statistics 103 Series Editrs: G. Casella S. Fienberg I. Olkin Fr further vlumes:

3

4 Gareth James Daniela Witten Trevr Hastie Rbert Tibshirani An Intrductin t Statistical Learning with Applicatins in R 123

5 Gareth James Department f Infrmatin and Operatins Management University f Suthern Califrnia Ls Angeles, CA, USA Trevr Hastie Department f Statistics Stanfrd University Stanfrd, CA, USA Daniela Witten Department f Bistatistics University f Washingtn Seattle, WA, USA Rbert Tibshirani Department f Statistics Stanfrd University Stanfrd, CA, USA ISSN X ISBN ISBN (ebk) DOI / Springer New Yrk Heidelberg Drdrecht Lndn Library f Cngress Cntrl Number: Springer Science+Business Media New Yrk 2013 This wrk is subject t cpyright. All rights are reserved by the Publisher, whether the whle r part f the material is cncerned, specifically the rights f translatin, reprinting, reuse f illustratins, recitatin, bradcasting, reprductin n micrfilms r in any ther physical way, and transmissin r infrmatin strage and retrieval, electrnic adaptatin, cmputer sftware, r by similar r dissimilar methdlgy nw knwn r hereafter develped. Exempted frm this legal reservatin are brief excerpts in cnnectin with reviews r schlarly analysis r material supplied specifically fr the purpse f being entered and executed n a cmputer system, fr exclusive use by the purchaser f the wrk. Duplicatin f this publicatin r parts theref is permitted nly under the prvisins f the Cpyright Law f the Publisher s lcatin, in its current versin, and permissin fr use must always be btained frm Springer. Permissins fr use may be btained thrugh RightsLink at the Cpyright Clearance Center. Vilatins are liable t prsecutin under the respective Cpyright Law. The use f general descriptive names, registered names, trademarks, service marks, etc. in this publicatin des nt imply, even in the absence f a specific statement, that such names are exempt frm the relevant prtective laws and regulatins and therefre free fr general use. While the advice and infrmatin in this bk are believed t be true and accurate at the date f publicatin, neither the authrs nr the editrs nr the publisher can accept any legal respnsibility fr any errrs r missins that may be made. The publisher makes n warranty, express r implied, with respect t the material cntained herein. Printed n acid-free paper Springer is part f Springer Science+Business Media (

6 T ur parents: Alisn and Michael James Chiara Nappi and Edward Witten Valerie and Patrick Hastie Vera and Sami Tibshirani and t ur families: Michael, Daniel, and Catherine Ari Samantha, Timthy, and Lynda Charlie, Ryan, Julie, and Cheryl

7

8 Preface Statistical learning refers t a set f tls fr mdeling and understanding cmplex datasets. It is a recently develped area in statistics and blends with parallel develpments in cmputer science and, in particular, machine learning. The field encmpasses many methds such as the lass and sparse regressin, classificatin and regressin trees, and bsting and supprt vectr machines. With the explsin f Big Data prblems, statistical learning has becme a very ht field in many scientific areas as well as marketing, finance, and ther business disciplines. Peple with statistical learning skills are in high demand. One f the first bks in this area The Elements f Statistical Learning (ESL) (Hastie, Tibshirani, and Friedman) was published in 2001, with a secnd editin in ESL has becme a ppular text nt nly in statistics but als in related fields. One f the reasns fr ESL s ppularity is its relatively accessible style. But ESL is intended fr individuals with advanced training in the mathematical sciences. An Intrductin t Statistical Learning (ISL) arse frm the perceived need fr a brader and less technical treatment f these tpics. In this new bk, we cver many f the same tpics as ESL, but we cncentrate mre n the applicatins f the methds and less n the mathematical details. We have created labs illustrating hw t implement each f the statistical learning methds using the ppular statistical sftware package R. These labs prvide the reader with valuable hands-n experience. This bk is apprpriate fr advanced undergraduates r master s students in statistics r related quantitative fields r fr individuals in ther vii

9 viii Preface disciplines wh wish t use statistical learning tls t analyze their data. It can be used as a textbk fr a curse spanning ne r tw semesters. We wuld like t thank several readers fr valuable cmments n preliminary drafts f this bk: Pallavi Basu, Alexandra Chuldechva, Patrick Danaher, Will Fithian, Luella Fu, Sam Grss, Max Grazier G Sell, Curtney Paulsn, Xingha Qia, Elisa Sheng, Nah Simn, Kean Ming Tan, and Xin Lu Tan. It s tugh t make predictins, especially abut the future. -Ygi Berra Ls Angeles, USA Seattle, USA Pal Alt, USA Pal Alt, USA Gareth James Daniela Witten Trevr Hastie Rbert Tibshirani

10 Cntents Preface vii 1 Intrductin 1 2 Statistical Learning What Is Statistical Learning? Why Estimate f? Hw D We Estimate f? The Trade-Off Between Predictin Accuracy and Mdel Interpretability Supervised Versus Unsupervised Learning Regressin Versus Classificatin Prblems Assessing Mdel Accuracy Measuring the Quality f Fit The Bias-Variance Trade-Off The Classificatin Setting Lab: Intrductin t R Basic Cmmands Graphics Indexing Data Lading Data Additinal Graphical and Numerical Summaries Exercises ix

11 x Cntents 3 Linear Regressin Simple Linear Regressin Estimating the Cefficients Assessing the Accuracy f the Cefficient Estimates Assessing the Accuracy f the Mdel Multiple Linear Regressin Estimating the Regressin Cefficients Sme Imprtant Questins Other Cnsideratins in the Regressin Mdel Qualitative Predictrs Extensins f the Linear Mdel Ptential Prblems The Marketing Plan Cmparisn f Linear Regressin with K-Nearest Neighbrs Lab: Linear Regressin Libraries Simple Linear Regressin Multiple Linear Regressin Interactin Terms Nn-linear Transfrmatins f the Predictrs Qualitative Predictrs Writing Functins Exercises Classificatin An Overview f Classificatin Why Nt Linear Regressin? Lgistic Regressin The Lgistic Mdel Estimating the Regressin Cefficients Making Predictins Multiple Lgistic Regressin Lgistic Regressin fr >2 Respnse Classes Linear Discriminant Analysis Using Bayes Therem fr Classificatin Linear Discriminant Analysis fr p = Linear Discriminant Analysis fr p> Quadratic Discriminant Analysis A Cmparisn f Classificatin Methds Lab: Lgistic Regressin, LDA, QDA, and KNN The Stck Market Data Lgistic Regressin Linear Discriminant Analysis

12 Cntents xi Quadratic Discriminant Analysis K-Nearest Neighbrs An Applicatin t Caravan Insurance Data Exercises Resampling Methds Crss-Validatin The Validatin Set Apprach Leave-One-Out Crss-Validatin k-fld Crss-Validatin Bias-Variance Trade-Off fr k-fld Crss-Validatin Crss-Validatin n Classificatin Prblems The Btstrap Lab: Crss-Validatin and the Btstrap The Validatin Set Apprach Leave-One-Out Crss-Validatin k-fld Crss-Validatin The Btstrap Exercises Linear Mdel Selectin and Regularizatin Subset Selectin Best Subset Selectin Stepwise Selectin Chsing the Optimal Mdel Shrinkage Methds Ridge Regressin The Lass Selecting the Tuning Parameter Dimensin Reductin Methds Principal Cmpnents Regressin Partial Least Squares Cnsideratins in High Dimensins High-Dimensinal Data What Ges Wrng in High Dimensins? Regressin in High Dimensins Interpreting Results in High Dimensins Lab 1: Subset Selectin Methds Best Subset Selectin Frward and Backward Stepwise Selectin Chsing Amng Mdels Using the Validatin Set Apprach and Crss-Validatin

13 xii Cntents 6.6 Lab 2: Ridge Regressin and the Lass Ridge Regressin The Lass Lab 3: PCR and PLS Regressin Principal Cmpnents Regressin Partial Least Squares Exercises Mving Beynd Linearity Plynmial Regressin Step Functins Basis Functins Regressin Splines Piecewise Plynmials Cnstraints and Splines The Spline Basis Representatin Chsing the Number and Lcatins f the Knts Cmparisn t Plynmial Regressin Smthing Splines An Overview f Smthing Splines Chsing the Smthing Parameter λ Lcal Regressin Generalized Additive Mdels GAMs fr Regressin Prblems GAMs fr Classificatin Prblems Lab: Nn-linear Mdeling Plynmial Regressin and Step Functins Splines GAMs Exercises Tree-Based Methds The Basics f Decisin Trees Regressin Trees Classificatin Trees Trees Versus Linear Mdels Advantages and Disadvantages f Trees Bagging, Randm Frests, Bsting Bagging Randm Frests Bsting Lab: Decisin Trees Fitting Classificatin Trees Fitting Regressin Trees

14 Cntents xiii Bagging and Randm Frests Bsting Exercises Supprt Vectr Machines Maximal Margin Classifier What Is a Hyperplane? Classificatin Using a Separating Hyperplane The Maximal Margin Classifier Cnstructin f the Maximal Margin Classifier The Nn-separable Case Supprt Vectr Classifiers Overview f the Supprt Vectr Classifier Details f the Supprt Vectr Classifier Supprt Vectr Machines Classificatin with Nn-linear Decisin Bundaries The Supprt Vectr Machine An Applicatin t the Heart Disease Data SVMs with Mre than Tw Classes One-Versus-One Classificatin One-Versus-All Classificatin Relatinship t Lgistic Regressin Lab: Supprt Vectr Machines Supprt Vectr Classifier Supprt Vectr Machine ROC Curves SVM with Multiple Classes Applicatin t Gene Expressin Data Exercises Unsupervised Learning The Challenge f Unsupervised Learning Principal Cmpnents Analysis What Are Principal Cmpnents? Anther Interpretatin f Principal Cmpnents Mre n PCA Other Uses fr Principal Cmpnents Clustering Methds K-Means Clustering Hierarchical Clustering Practical Issues in Clustering Lab 1: Principal Cmpnents Analysis

15 xiv Cntents 10.5 Lab 2: Clustering K-Means Clustering Hierarchical Clustering Lab 3: NCI60 Data Example PCA n the NCI60 Data Clustering the Observatins f the NCI60 Data Exercises Index 419

16 1 Intrductin An Overview f Statistical Learning Statistical learning refers t a vast set f tls fr understanding data. These tls can be classified as supervised r unsupervised. Bradly speaking, supervised statistical learning invlves building a statistical mdel fr predicting, r estimating, an utput basednnermreinputs. Prblemsf this nature ccur in fields as diverse as business, medicine, astrphysics, and public plicy. With unsupervised statistical learning, there are inputs but n supervising utput; nevertheless we can learn relatinships and structure frm such data. T prvide an illustratin f sme applicatins f statistical learning, we briefly discuss three real-wrld data sets that are cnsidered in this bk. Wage Data In this applicatin (which we refer t as the Wage data set thrughut this bk), we examine a number f factrs that relate t wages fr a grup f males frm the Atlantic regin f the United States. In particular, we wish t understand the assciatin between an emplyee s age and educatin, as well as the calendar year, n his wage. Cnsider, fr example, the left-hand panel f Figure 1.1, which displays wage versus age fr each f the individuals in the data set. There is evidence that wage increases with age but then decreases again after apprximately age 60. The blue line, which prvides an estimate f the average wage fr a given age, makes this trend clearer. G. James et al., An Intrductin t Statistical Learning: with Applicatins in R, Springer Texts in Statistics 103, DOI / , Springer Science+Business Media New Yrk

17 2 1. Intrductin Wage Wage Wage Age Year Educatin Level FIGURE 1.1. Wage data, which cntains incme survey infrmatin fr males frm the central Atlantic regin f the United States. Left: wage as a functin f age. On average, wage increases with age until abut 60 years f age, at which pint it begins t decline. Center: wage as a functin f year. Thereisaslw but steady increase f apprximately $10,000 in the average wage between 2003 and Right: Bxplts displaying wage as a functin f educatin, with1 indicating the lwest level (n high schl diplma) and 5 the highest level (an advanced graduate degree). On average, wage increases with the level f educatin. Givenanemplyee sage, we can use this curve t predict his wage. Hwever, it is als clear frm Figure 1.1 that there is a significant amunt f variability assciated with this average value, and s age alne is unlikely t prvide an accurate predictin f a particular man s wage. We als have infrmatin regarding each emplyee s educatin level and the year in which the wage was earned. The center and right-hand panels f Figure 1.1, which display wage as a functin f bth year and educatin, indicate that bth f these factrs are assciated with wage. Wages increase by apprximately $10,000, in a rughly linear (r straight-line) fashin, between 2003 and 2009, thugh this rise is very slight relative t the variability in the data. Wages are als typically greater fr individuals with higher educatin levels: men with the lwest educatin level (1) tend t have substantially lwer wages than thse with the highest educatin level (5). Clearly, the mst accurate predictin f a given man s wage will be btained by cmbining his age, his educatin, and the year. In Chapter 3, we discuss linear regressin, which can be used t predict wage frm this data set. Ideally, we shuld predict wage in a way that accunts fr the nn-linear relatinship between wage and age. In Chapter 7, we discuss a class f appraches fr addressing this prblem. Stck Market Data The Wage data invlves predicting a cntinuus r quantitative utput value. This is ften referred t as a regressin prblem. Hwever, in certain cases we may instead wish t predict a nn-numerical value that is, a categrical

18 1. Intrductin 3 Yesterday Tw Days Previus Three Days Previus Percentage change in S&P Percentage change in S&P Percentage change in S&P Dwn Up Tday s Directin Dwn Up Tday s Directin Dwn Up Tday s Directin FIGURE 1.2. Left: Bxplts f the previus day s percentage change in the S&P index fr the days fr which the market increased r decreased, btained frm the Smarket data. Center and Right: Same as left panel, but the percentage changes fr 2 and 3 days previus are shwn. r qualitative utput. Fr example, in Chapter 4 we examine a stck market data set that cntains the daily mvements in the Standard & Pr s 500 (S&P) stck index ver a 5-year perid between 2001 and We refer t this as the Smarket data. The gal is t predict whether the index will increase r decrease n a given day using the past 5 days percentage changes in the index. Here the statistical learning prblem des nt invlve predicting a numerical value. Instead it invlves predicting whether a given day s stck market perfrmance will fall int the Up bucket r the Dwn bucket. This is knwn as a classificatin prblem. A mdel that culd accurately predict the directin in which the market will mve wuld be very useful! The left-hand panel f Figure 1.2 displays tw bxplts f the previus day s percentage changes in the stck index: ne fr the 648 days fr which the market increased n the subsequent day, and ne fr the 602 days fr which the market decreased. The tw plts lk almst identical, suggesting that there is n simple strategy fr using yesterday s mvement in the S&P t predict tday s returns. The remaining panels, which display bxplts fr the percentage changes 2 and 3 days previus t tday, similarly indicate little assciatin between past and present returns. Of curse, this lack f pattern is t be expected: in the presence f strng crrelatins between successive days returns, ne culd adpt a simple trading strategy t generate prfits frm the market. Nevertheless, in Chapter 4, we explre these data using several different statistical learning methds. Interestingly, there are hints f sme weak trends in the data that suggest that, at least fr this 5-year perid, it is pssible t crrectly predict the directin f mvement in the market apprximately 60% f the time (Figure 1.3).

19 4 1. Intrductin Predicted Prbability Dwn Up Tday s Directin FIGURE 1.3. We fit a quadratic discriminant analysis mdel t the subset f the Smarket data crrespnding t the time perid, and predicted the prbability f a stck market decrease using the 2005 data. On average, the predicted prbability f decrease is higher fr the days in which the market des decrease. Based n these results, we are able t crrectly predict the directin f mvement in the market 60% f the time. Gene Expressin Data The previus tw applicatins illustrate data sets with bth input and utput variables. Hwever, anther imprtant class f prblems invlves situatins in which we nly bserve input variables, with n crrespnding utput. Fr example, in a marketing setting, we might have demgraphic infrmatin fr a number f current r ptential custmers. We may wish t understand which types f custmers are similar t each ther by gruping individuals accrding t their bserved characteristics. This is knwn as a clustering prblem. Unlike in the previus examples, here we are nt trying t predict an utput variable. We devte Chapter 10 t a discussin f statistical learning methds fr prblems in which n natural utput variable is available. We cnsider the NCI60 data set, which cnsists f 6,830 gene expressin measurements fr each f 64 cancer cell lines. Instead f predicting a particular utput variable, we are interested in determining whether there are grups, r clusters, amng the cell lines based n their gene expressin measurements. This is a difficult questin t address, in part because there are thusands f gene expressin measurements per cell line, making it hard t visualize the data. The left-hand panel f Figure 1.4 addresses this prblem by representing each f the 64 cell lines using just tw numbers, Z 1 and Z 2.These are the first tw principal cmpnents f the data, which summarize the 6, 830 expressin measurements fr each cell line dwn t tw numbers r dimensins. While it is likely that this dimensin reductin has resulted in

20 1. Intrductin 5 Z Z Z Z 1 FIGURE 1.4. Left: Representatin f the NCI60 gene expressin data set in a tw-dimensinal space, Z 1 and Z 2. Each pint crrespnds t ne f the 64 cell lines. There appear t be fur grups f cell lines, which we have represented using different clrs. Right: Same as left panel except that we have represented each f the 14 different types f cancer using a different clred symbl. Cell lines crrespnding t the same cancer type tend t be nearby in the tw-dimensinal space. sme lss f infrmatin, it is nw pssible t visually examine the data fr evidence f clustering. Deciding n the number f clusters is ften a difficult prblem. But the left-hand panel f Figure 1.4 suggests at least fur grups f cell lines, which we have represented using separate clrs. We can nw examine the cell lines within each cluster fr similarities in their types f cancer, in rder t better understand the relatinship between gene expressin levels and cancer. In this particular data set, it turns ut that the cell lines crrespnd t 14 different types f cancer. (Hwever, this infrmatin was nt used t create the left-hand panel f Figure 1.4.) The right-hand panel f Figure 1.4 is identical t the left-hand panel, except that the 14 cancer types are shwn using distinct clred symbls. There is clear evidence that cell lines with the same cancer type tend t be lcated near each ther in this tw-dimensinal representatin. In additin, even thugh the cancer infrmatin was nt used t prduce the left-hand panel, the clustering btained des bear sme resemblance t sme f the actual cancer types bserved in the right-hand panel. This prvides sme independent verificatin f the accuracy f ur clustering analysis. A Brief Histry f Statistical Learning Thugh the term statistical learning is fairly new, many f the cncepts that underlie the field were develped lng ag. At the beginning f the nineteenth century, Legendre and Gauss published papers n the methd

21 6 1. Intrductin f least squares, which implemented the earliest frm f what is nw knwn as linear regressin. The apprach was first successfully applied t prblems in astrnmy. Linear regressin is used fr predicting quantitative values, such as an individual s salary. In rder t predict qualitative values, such as whether a patient survives r dies, r whether the stck market increases r decreases, Fisher prpsed linear discriminant analysis in In the 1940s, varius authrs put frth an alternative apprach, lgistic regressin. In the early 1970s, Nelder and Wedderburn cined the term generalized linear mdels fr an entire class f statistical learning methds that include bth linear and lgistic regressin as special cases. By the end f the 1970s, many mre techniques fr learning frm data were available. Hwever, they were almst exclusively linear methds, because fitting nn-linear relatinships was cmputatinally infeasible at the time. By the 1980s, cmputing technlgy had finally imprved sufficiently that nn-linear methds were n lnger cmputatinally prhibitive. In mid 1980s Breiman, Friedman, Olshen and Stne intrduced classificatin and regressin trees, and were amng the first t demnstrate the pwer f a detailed practical implementatin f a methd, including crss-validatin fr mdel selectin. Hastie and Tibshirani cined the term generalized additive mdels in 1986 fr a class f nn-linear extensins t generalized linear mdels, and als prvided a practical sftware implementatin. Since that time, inspired by the advent f machine learning and ther disciplines, statistical learning has emerged as a new subfield in statistics, fcused n supervised and unsupervised mdeling and predictin. In recent years, prgress in statistical learning has been marked by the increasing availability f pwerful and relatively user-friendly sftware, such as the ppular and freely available R system. This has the ptential t cntinue the transfrmatin f the field frm a set f techniques used and develped by statisticians and cmputer scientists t an essential tlkit fr a much brader cmmunity. This Bk The Elements f Statistical Learning (ESL) by Hastie, Tibshirani, and Friedman was first published in Since that time, it has becme an imprtant reference n the fundamentals f statistical machine learning. Its success derives frm its cmprehensive and detailed treatment f many imprtant tpics in statistical learning, as well as the fact that (relative t many upper-level statistics textbks) it is accessible t a wide audience. Hwever, the greatest factr behind the success f ESL has been its tpical nature. At the time f its publicatin, interest in the field f statistical

22 1. Intrductin 7 learning was starting t explde. ESL prvided ne f the first accessible and cmprehensive intrductins t the tpic. Since ESL was first published, the field f statistical learning has cntinued t flurish. The field s expansin has taken tw frms. The mst bvius grwth has invlved the develpment f new and imprved statistical learning appraches aimed at answering a range f scientific questins acrss a number f fields. Hwever, the field f statistical learning has als expanded its audience. In the 1990s, increases in cmputatinal pwer generated a surge f interest in the field frm nn-statisticians wh were eager t use cutting-edge statistical tls t analyze their data. Unfrtunately, the highly technical nature f these appraches meant that the user cmmunity remained primarily restricted t experts in statistics, cmputer science, and related fields with the training (and time) t understand and implement them. In recent years, new and imprved sftware packages have significantly eased the implementatin burden fr many statistical learning methds. At the same time, there has been grwing recgnitin acrss a number f fields, frm business t health care t genetics t the scial sciences and beynd, that statistical learning is a pwerful tl with imprtant practical applicatins. As a result, the field has mved frm ne f primarily academic interest t a mainstream discipline, with an enrmus ptential audience. This trend will surely cntinue with the increasing availability f enrmus quantities f data and the sftware t analyze it. The purpse f An Intrductin t Statistical Learning (ISL) is t facilitate the transitin f statistical learning frm an academic t a mainstream field. ISL is nt intended t replace ESL, which is a far mre cmprehensive text bth in terms f the number f appraches cnsidered and the depth t which they are explred. We cnsider ESL t be an imprtant cmpanin fr prfessinals (with graduate degrees in statistics, machine learning, r related fields) wh need t understand the technical details behind statistical learning appraches. Hwever, the cmmunity f users f statistical learning techniques has expanded t include individuals with a wider range f interests and backgrunds. Therefre, we believe that there is nw a place fr a less technical and mre accessible versin f ESL. In teaching these tpics ver the years, we have discvered that they are f interest t master s and PhD students in fields as disparate as business administratin, bilgy, and cmputer science, as well as t quantitativelyriented upper-divisin undergraduates. It is imprtant fr this diverse grup t be able t understand the mdels, intuitins, and strengths and weaknesses f the varius appraches. But fr this audience, many f the technical details behind statistical learning methds, such as ptimizatin algrithms and theretical prperties, are nt f primary interest. We believe that these students d nt need a deep understanding f these aspects in rder t becme infrmed users f the varius methdlgies, and