Introduction to Machine Learning

Transcription

1 Introduction to Machine Learning Isabelle Guyon

2 What is Machine Learning? Learning algorithm Trained machine TRAINING DATA Answer Query

3 What for? Classification Time series prediction Regression Clustering

4 Some Learning Machines Linear models Kernel methods Neural networks Decision trees

5 Applications training examples Ecology System diagnosis Market Analysis OCR HWR Machine Vision Text Categorization Bioinformatics inputs

6 Banking / Telecom / Retail Identify: Prospective customers Dissatisfied customers Good customers Bad payers Obtain: More effective advertising Less credit risk Fewer fraud Decreased churn rate

7 Biomedical / Biometrics Medicine: Screening Diagnosis and prognosis Drug discovery Security: Face recognition Signature / fingerprint / iris verification DNA fingerprinting 6

8 Computer / Internet Computer interfaces: Troubleshooting wizards Handwriting and speech Brain waves Internet Hit ranking Spam filtering Text categorization Text translation Recommendation 7

9 Challenges training examples 10 5 Sylva Ada NIPS 2003 & WCCI Dexter, Nova Madelon Gisette Gina Arcene, Dorothea, Hiva inputs

10 Ten Classification Tasks ARCENE DEXTER DOROTHEA GISETTE MADELON Test BER (%) ADA GINA HIVA NOVA SYLVA

11 Challenge Winning Methods BER/<BER> Linear /Kernel Neural Nets Trees /RF Naïve Bayes Gisette (HWR) Gina (HWR) Dexter (Text) Nova (Text) Madelon (Artificial) Arcene (Spectral) Dorothea (Pharma) Hiva (Pharma) Ada (Marketing) Sylva (Ecology)

12 Conventions n X={x ij } y ={y x m j } i α w

13 Learning problem Data matrix: X m lines = patterns (data points, examples): samples, patients, documents, images, n columns = features: (attributes, input variables): genes, proteins, words, pixels, Colon cancer, Alon et al 1999 Unsupervised learning Is there structure in data? Supervised learning Predict an outcome y.

14 Linear Models f(x) = w x +b = Σ j=1:n w j x j +b Linearity in the parameters, NOT in the input components. f(x) = w Φ(x) +b = Σ j w j φ j (x) +b (Perceptron) f(x) = Σ i=1:m α i k(x i,x) +b (Kernel method)

15 Artificial Neurons x 1 Cell potential w 1 x 2 w 2 Σ f(x) Activation of other neurons x n 1 w n b Synapses Dendrites Axon Activation function McCulloch and Pitts, 1943 f(x) = w x + b

16 Linear Decision Boundary hyperplane 0.5 x x 3 X x x X xx

17 Perceptron x 1 φ 1 (x) Rosenblatt, 1957 x 2 φ 2 (x) w 1 w 2 Σ f(x) x n φ N (x) 1 w N b f(x) = w Φ(x) + b

18 NL Decision Boundary x Hs.7780 x x 1 x 2 Hs x 1 Hs

19 Kernel Method x 1 k(x 1,x) Potential functions, Aizerman et al 1964 x 2 k(x 2,x) α 1 α 2 Σ x n k(x m,x) 1 α m b f(x) = Σ i α i k(x i,x) + b k(.,. ) is a similarity measure or kernel.

20 Hebb s Rule w j w j + y i x ij Activation of another neuron x j w j Σ y Axon Dendrite Synapse Link to Naïve Bayes

21 Kernel Trick (for Hebb s rule) Hebb s rule for the Perceptron: w = Σ i y i Φ(x i ) f(x) = w Φ(x) = Σ i y i Φ(x i ) Φ(x) Define a dot product: k(x i,x) = Φ(x i ) Φ(x) f(x) = Σ i y i k(x i,x)

22 Kernel Trick (general) f(x) = Σ i α i k(x i, x) k(x i, x) = Φ(x i ) Φ(x) Dual forms f(x) = w Φ(x) w = Σ i α i Φ(x i )

23 What is a Kernel? A kernel is: a similarity measure a dot product in some feature space: k(s, t) = Φ(s) Φ(t) But we do not need to know the Φ representation. Examples: k(s, t) = exp(- s-t 2 /σ 2 ) k(s, t) = (s t) q Gaussian kernel Polynomial kernel

24 Multi-Layer Perceptron Back-propagation, Rumelhart et al, 1986 Σ x j Σ Σ internal latent variables hidden units

25 Chessboard Problem

26 Tree Classifiers CART (Breiman, 1984) or C4.5 (Quinlan, 1993) f 2 All the data Choose f 1 Choose f 2 f 1 At each step, choose the feature that reduces entropy most. Work towards node purity.

27 Linear discriminant Iris Data (Fisher, 1936) Figure from Norbert Jankowski and Krzysztof Grabczewski Tree classifier setosa virginica versicolor Gaussian mixture Kernel method (SVM)

28 Fit / Robustness Tradeoff x 2 x 2 x 1 x 1 15

29 Performance evaluation f(x) = 0 f(x) < 0 f(x) < 0 x 2 x 2 f(x) = 0 f(x) > 0 f(x) > 0 x 1 x 1

30 Performance evaluation f(x) = -1 f(x) < -1 f(x) < -1 x 2 x 2 f(x) = -1 f(x) > -1 f(x) > -1 x 1 x 1

31 Performance evaluation f(x) = 1 f(x) < 1 f(x) < 1 x 2 x 2 f(x) = 1 f(x) > 1 f(x) > 1 x 1 x 1

32 ROC Curve For a given threshold on f(x), you get a point on the ROC curve. 100% Ideal ROC curve Actual ROC Positive class success rate (hit rate, sensitivity) Random ROC negative class success rate (false alarm rate, 1-specificity) 100%

33 ROC Curve For a given threshold on f(x), you get a point on the ROC curve. 100% Positive class success rate (hit rate, sensitivity) Ideal ROC curve (AUC=1) Actual ROC Random ROC (AUC=0.5) 0 0 AUC negative class success rate (false alarm rate, 1-specificity) 100%

34 Lift Curve Customers ranked according to f(x); selection of the top ranking customers. Gini = M O Gini=2 AUC-1 0 Gini 1 100% Hit rate = Frac. good customers select. 0 O Ideal Lift Actual Lift M Random lift Fraction of customers selected 100%

35 Performance Assessment Cost matrix Truth: y Predictions: F(x) Class -1 Class +1 Class -1 tn fp Class +1 fn tp Total Class+1 /Total rej=tn+fn sel=fp+tp Precision = tp/sel Total neg=tn+fp pos=fn+tp Class +1 / Total False alarm = fp/neg Hit rate = tp/pos m=tn+fp Frac. selected = sel/m +fn+tp False alarm rate = type I errate = 1-specificity Hit rate = 1-type II errate = sensitivity = recall = test power Compare F(x) = sign(f(x)) to the target y, and report: Error rate = (fn + fp)/m {Hit rate, False alarm rate} or {Hit rate, Precision} or {Hit rate, Frac.selected} Balanced error rate (BER) = (fn/pos + fp/neg)/2 = 1 (sensitivity+specificity)/2 F measure = 2 precision.recall/(precision+recall) Vary the decision threshold θ in F(x) = sign(f(x)+θ), and plot: ROC curve: Hit rate vs. False alarm rate Lift curve: Hit rate vs. Fraction selected Precision/recall curve: Hit rate vs. Precision

36 What is a Risk Functional? A function of the parameters of the learning machine, assessing how much it is expected to fail on a given task. Examples: Classification: Error rate: (1/m) Σ i=1:m 1(F(x i ) y i ) 1- AUC (Gini Index = 2 AUC-1) Regression: Mean square error: (1/m) Σ i=1:m (f(x i )-y i ) 2

37 How to train? Define a risk functional R[f(x,w)] Optimize it w.r.t. w (gradient descent, mathematical programming, simulated annealing, genetic algorithms, etc.) R[f(x,w)] Parameter space (w) w* ( to be continued in the next lecture)

38 How to Train? Define a risk functional R[f(x,w)] Find a method to optimize it, typically gradient descent w j w j - η R/ w j or any optimization method (mathematical programming, simulated annealing, genetic algorithms, etc.) ( to be continued in the next lecture)

39 Summary With linear threshold units ( neurons ) we can build: Linear discriminant (including Naïve Bayes) Kernel methods Neural networks Decision trees The architectural hyper-parameters may include: The choice of basis functions φ (features) The kernel The number of units Learning means fitting: Parameters (weights) Hyper-parameters Be aware of the fit vs. robustness tradeoff

40 Want to Learn More? Pattern Classification, R. Duda, P. Hart, and D. Stork. Standard pattern recognition textbook. Limited to classification problems. Matlab code. The Elements of statistical Learning: Data Mining, Inference, and Prediction. T. Hastie, R. Tibshirani, J. Friedman, Standard statistics textbook. Includes all the standard machine learning methods for classification, regression, clustering. R code. Linear Discriminants and Support Vector Machines, I. Guyon and D. Stork, In Smola et al Eds. Advances in Large Margin Classiers. Pages , MIT Press, Feature Extraction: Foundations and Applications. I. Guyon et al, Eds. Book for practitioners with datasets of NIPS 2003 challenge, tutorials, best performing methods, Matlab code, teaching material.