Introduction to Machine Learning

Transcription

1 Introduction to Machine Learning Third Edition Ethem Alpaydın The MIT Press Cambridge, Massachusetts London, England

2 2014 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. For information about special quantity discounts, please Typeset in 10/13 Lucida Bright by the author using L A T E X2ε. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Information Alpaydin, Ethem. Introduction to machine learning / Ethem Alpaydin 3rd ed. p. cm. Includes bibliographical references and index. ISBN (hardcover : alk. paper) 1. Machine learning. I. Title Q325.5.A dc CIP

3 Brief Contents 1 Introduction 1 2 Supervised Learning 21 3 Bayesian Decision Theory 49 4 Parametric Methods 65 5 Multivariate Methods 93 6 Dimensionality Reduction Clustering Nonparametric Methods Decision Trees Linear Discrimination Multilayer Perceptrons Local Models Kernel Machines Graphical Models Hidden Markov Models Bayesian Estimation Combining Multiple Learners Reinforcement Learning Design and Analysis of Machine Learning Experiments 547 A Probability 593

4

5 Contents Preface Notations xvii xxi 1 Introduction What Is Machine Learning? Examples of Machine Learning Applications Learning Associations Classification Regression Unsupervised Learning Reinforcement Learning Notes Relevant Resources Exercises References 20 2 Supervised Learning Learning a Class from Examples Vapnik-Chervonenkis Dimension Probably Approximately Correct Learning Noise Learning Multiple Classes Regression Model Selection and Generalization Dimensions of a Supervised Machine Learning Algorithm Notes 42

6 viii Contents 2.10 Exercises References 47 3 Bayesian Decision Theory Introduction Classification Losses and Risks Discriminant Functions Association Rules Notes Exercises References 64 4 Parametric Methods Introduction Maximum Likelihood Estimation Bernoulli Density Multinomial Density Gaussian (Normal) Density Evaluating an Estimator: Bias and Variance The Bayes Estimator Parametric Classification Regression Tuning Model Complexity: Bias/Variance Dilemma Model Selection Procedures Notes Exercises References 90 5 Multivariate Methods Multivariate Data Parameter Estimation Estimation of Missing Values Multivariate Normal Distribution Multivariate Classification Tuning Complexity Discrete Features Multivariate Regression Notes Exercises 112

7 Contents ix 5.11 References Dimensionality Reduction Introduction Subset Selection Principal Component Analysis Feature Embedding Factor Analysis Singular Value Decomposition and Matrix Factorization Multidimensional Scaling Linear Discriminant Analysis Canonical Correlation Analysis Isomap Locally Linear Embedding Laplacian Eigenmaps Notes Exercises References Clustering Introduction Mixture Densities k-means Clustering Expectation-Maximization Algorithm Mixtures of Latent Variable Models Supervised Learning after Clustering Spectral Clustering Hierarchical Clustering Choosing the Number of Clusters Notes Exercises References Nonparametric Methods Introduction Nonparametric Density Estimation Histogram Estimator Kernel Estimator k-nearest Neighbor Estimator Generalization to Multivariate Data 192

8 x Contents 8.4 Nonparametric Classification Condensed Nearest Neighbor Distance-Based Classification Outlier Detection Nonparametric Regression: Smoothing Models Running Mean Smoother Kernel Smoother Running Line Smoother How to Choose the Smoothing Parameter Notes Exercises References Decision Trees Introduction Univariate Trees Classification Trees Regression Trees Pruning Rule Extraction from Trees Learning Rules from Data Multivariate Trees Notes Exercises References Linear Discrimination Introduction Generalizing the Linear Model Geometry of the Linear Discriminant Two Classes Multiple Classes Pairwise Separation Parametric Discrimination Revisited Gradient Descent Logistic Discrimination Two Classes Multiple Classes Discrimination by Regression 257

9 Contents xi 10.9 Learning to Rank Notes Exercises References Multilayer Perceptrons Introduction Understanding the Brain Neural Networks as a Paradigm for Parallel Processing The Perceptron Training a Perceptron Learning Boolean Functions Multilayer Perceptrons MLP as a Universal Approximator Backpropagation Algorithm Nonlinear Regression Two-Class Discrimination Multiclass Discrimination Multiple Hidden Layers Training Procedures Improving Convergence Overtraining Structuring the Network Hints Tuning the Network Size Bayesian View of Learning Dimensionality Reduction Learning Time Time Delay Neural Networks Recurrent Networks Deep Learning Notes Exercises References Local Models Introduction Competitive Learning 318

10 xii Contents Online k-means Adaptive Resonance Theory Self-Organizing Maps Radial Basis Functions Incorporating Rule-Based Knowledge Normalized Basis Functions Competitive Basis Functions Learning Vector Quantization The Mixture of Experts Cooperative Experts Competitive Experts Hierarchical Mixture of Experts Notes Exercises References Kernel Machines Introduction Optimal Separating Hyperplane The Nonseparable Case: Soft Margin Hyperplane ν-svm Kernel Trick Vectorial Kernels Defining Kernels Multiple Kernel Learning Multiclass Kernel Machines Kernel Machines for Regression Kernel Machines for Ranking One-Class Kernel Machines Large Margin Nearest Neighbor Classifier Kernel Dimensionality Reduction Notes Exercises References Graphical Models Introduction Canonical Cases for Conditional Independence Generative Models 396

11 Contents xiii 14.4 d-separation Belief Propagation Chains Trees Polytrees Junction Trees Undirected Graphs: Markov Random Fields Learning the Structure of a Graphical Model Influence Diagrams Notes Exercises References Hidden Markov Models Introduction Discrete Markov Processes Hidden Markov Models Three Basic Problems of HMMs Evaluation Problem Finding the State Sequence Learning Model Parameters Continuous Observations The HMM as a Graphical Model Model Selection in HMMs Notes Exercises References Bayesian Estimation Introduction Bayesian Estimation of the Parameters of a Discrete Distribution K>2 States: Dirichlet Distribution K = 2 States: Beta Distribution Bayesian Estimation of the Parameters of a Gaussian Distribution Univariate Case: Unknown Mean, Known Variance 451

12 xiv Contents Univariate Case: Unknown Mean, Unknown Variance Multivariate Case: Unknown Mean, Unknown Covariance Bayesian Estimation of the Parameters of a Function Regression Regression with Prior on Noise Precision The Use of Basis/Kernel Functions Bayesian Classification Choosing a Prior Bayesian Model Comparison Bayesian Estimation of a Mixture Model Nonparametric Bayesian Modeling Gaussian Processes Dirichlet Processes and Chinese Restaurants Latent Dirichlet Allocation Beta Processes and Indian Buffets Notes Exercises References Combining Multiple Learners Rationale Generating Diverse Learners Model Combination Schemes Voting Error-Correcting Output Codes Bagging Boosting The Mixture of Experts Revisited Stacked Generalization Fine-Tuning an Ensemble Choosing a Subset of the Ensemble Constructing Metalearners Cascading Notes Exercises References 513

13 Contents xv 18 Reinforcement Learning Introduction Single State Case: K-Armed Bandit Elements of Reinforcement Learning Model-Based Learning Value Iteration Policy Iteration Temporal Difference Learning Exploration Strategies Deterministic Rewards and Actions Nondeterministic Rewards and Actions Eligibility Traces Generalization Partially Observable States The Setting Example: The Tiger Problem Notes Exercises References Design and Analysis of Machine Learning Experiments Introduction Factors, Response, and Strategy of Experimentation Response Surface Design Randomization, Replication, and Blocking Guidelines for Machine Learning Experiments Cross-Validation and Resampling Methods K-Fold Cross-Validation Cross-Validation Bootstrapping Measuring Classifier Performance Interval Estimation Hypothesis Testing Assessing a Classification Algorithm s Performance Binomial Test Approximate Normal Test t Test Comparing Two Classification Algorithms McNemar s Test 573

14 xvi Contents K-Fold Cross-Validated Paired t Test cv Paired t Test cv Paired F Test Comparing Multiple Algorithms: Analysis of Variance Comparison over Multiple Datasets Comparing Two Algorithms Multiple Algorithms Multivariate Tests Comparing Two Algorithms Comparing Multiple Algorithms Notes Exercises References 590 A Probability 593 A.1 Elements of Probability 593 A.1.1 Axioms of Probability 594 A.1.2 Conditional Probability 594 A.2 Random Variables 595 A.2.1 Probability Distribution and Density Functions 595 A.2.2 Joint Distribution and Density Functions 596 A.2.3 Conditional Distributions 596 A.2.4 Bayes Rule 597 A.2.5 Expectation 597 A.2.6 Variance 598 A.2.7 Weak Law of Large Numbers 599 A.3 Special Random Variables 599 A.3.1 Bernoulli Distribution 599 A.3.2 Binomial Distribution 600 A.3.3 Multinomial Distribution 600 A.3.4 Uniform Distribution 600 A.3.5 Normal (Gaussian) Distribution 601 A.3.6 Chi-Square Distribution 602 A.3.7 t Distribution 603 A.3.8 F Distribution 603 A.4 References 603 Index 605