Introduction to Machine Learning. CAP5610: Machine Learning Instructor: Guo-Jun Qi

Transcription

1 Introduction to Machine Learning CAP5610: Machine Learning Instructor: Guo-Jun Qi

2 Today s topics Course information, textbooks and grading policy Introduction to Machine Learning Simple machine algorithm

3 Textbook Required: Pattern Recognition and Machine Learning, C. Bishop, Springer, Free download of ebook at Springer (check at for education purpose Recommended: Machine Learning, T. Mitchell, McGraw-Hill, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition, T. Hastie, R. Tibshirani, J. Friedman, Springer, Probabilistic Graphical Models: Principles and Techniques, D. Koller, and N. Friedman, The MIT Press, 2009.

4 Grading policy Machine Problems and Homework (60%) Report (solution, setting, test protocol, and result) Source code (matlab, python, c/c++, java etc.) due before the class on the deadline day Mid-term exam (20%) Final project (20%) Survey on a machine learning topic, or A proposed machine learning problem Technical report in an academic paper format (intro, formulation, experiment design and result, conclusion, references) TBD: A presentation as in conference (15 min presentation + 5 min Q&A)

5 Late submission Late 24 hours 50% Late 48 hours 25% More than 48 hours 0%

6 What will you learn from this course? A glance at syllabus Introduction to machine learning: feature extraction, examples of machine learning problems, training and test protocol, datasets, KNN Naive Bayesian classifier Linear regression and classification Support vector machine and kernel method Neural Networks and back propagation Unsupervised learning problems: clustering, PCA, LDA, CCA etc. Model fitting and EM algorithm Graphical model: Bayesian networks, Markov random fields, approximation inference, variational method, sampling, loopy belief propagation Advanced topics: matrix factorization, metric learning, latent models, online learning, active learning, latent models, sparse coding, nonparametric Bayesian model etc. (will cover as many as possible if time allows)

7 What s machine learning? A supervised learning task Data (patient records) A prediction task: will the patient suffer an emergent C- section? Examples extracted from T. Mitchell s slides

8 Past experiences You already know some emergent (non-emergent) C-section cases Positive example Emergent C-section Negative example Non-emergent C-section

9 Machine learning aims at Learning knowledge from the past experiences, and predicting on the future cases Training set of examples: Past experience (labeled examples) Test set of examples: future cases to predict on (unlabeled examples) A model is trained from the training set, which summarizes the knowledge from the past experience

10 An example of model Rule-based model for predicting emergent C-section Applying the model to predict on the future case

11 Input and output Input: training set Training set = {(x i, y i ) x i is the data, y i is the label} Output: Model can be viewed as a function, which maps data x to label y y=h(x): X Y The set of all possible functions constitute hypotheses H={h h:x->y} test set = {(x j,?) x j the data whose label will be predicted by the trained model}

12 Oracle function Assume we have an oracle function h o which always outputs a correct prediction on an input data x Machine learning algorithms aim to find a function h from a set of hypotheses H to approximate this oracle function as well as possible h = min h H Ε x D err h(x, h o (x where E denotes the expectation, D is the distribution of all possible examples in the real world, and err is a function measuring the discrepancy between the outputs from h and oracle function h o.

13 What s the challenge? Ideal objective of machine learning algorithm h = min h H Ε x D err h(x, h o (x We do not know what h o will output on the examples out of training set Solution: using training set to approximate the objective Ε x D err h(x, h o (x n 1 n i=1 err h(x i, y i

14 How good is the approximation? Using the sample mean to approximate the distribution mean Ε x D err h(x, h o (x n 1 n i=1 err h(x i, y i The law of large number: the sample mean will approach to the distribution mean as n goes to infinity. (asymptotically) Learning theory: quantifying the discrepancy between sample mean and distribution mean of error under a given number of training examples

15 Some notes We do not require the oracle function belongs to hypothesis set H The training set may have noise the output y of a input x may not be correct y h o (x)

16 Choose error function Depending on the nature of output variable Discrete value {0,1,,C}: err(h(x),y) is 0 if h(x) and y is the same, or 1 otherwise Continuous value, squared difference err(h(x),y)= (h(x)-y)^2 Vector of continuous number, squared Euclidean distance err(h(x),y)

17 More examples Hand-written digit recognition (zip code) MNIST (Mixed National Institute of Standards and Technology) dataset 60,000 training examples: written by American census Bureau employees 10,000 test examples: written by American high school students recognizing the digits from 0 to 9. How good is machine learning algorithm on this task? Best performance: 0.27% test error, closer to human performance

18 How to represent the data in computer? Data representation that can be processed by computer. A table of attributes Integer: 23 Boolean: No Boolean: No Boolean: YES Boolean: No Enumeration: Abnormal Boolean: No

19 Feature extraction Data representation for hand-written digits Table of attributes Pixel value: 0 Pixel value: 255 Pixel value: 255 Pixel value: 255 Pixel value: 0 Pixel value: 0 Pixel value: 0 Feature vectors

20 Feature vector space Each point in the space represents a feature vector whose entry is an attribute of the data. Feature vector space A feature vector Integer: 23 Boolean: No Boolean: No Boolean: YES Boolean: No Enumeration: Abnormal Boolean: No

21 Principle of feature extraction Extracting features that are relevant to the defined task MNIST pixel values in the image Emergent C-section age, first pregnant, anemia, Domain knowledge Image processing Medicine

22 What machine learning cannot do Garbage in garbage out All features are completely irrelevant to the task, machine learning can do nothing for you. Good features play the key role in machine learning Domain knowledge will factor in the performance a lot.

23 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification Spam detection Input: image Output: face location Where is the face? From CMU face detection project

24 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification Spam detection Input: face Output: identity Whom is the face? From yale face dateset

25 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification Spam detection Localize and classify objects in image From PASCAL VOC 2012 dataset

26 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification Spam detection From speech to text Input: cepstral coefficients Output: words/sentences

27 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification spam detection Input: webpage caption, URL, keywords, incoming/outgoing links Output: webpage category (company, personal, university)

28 Other applications Face detection & recognition Object detection & recognition Speech recognition Webpage classification spam detection Input: sender, length, keywords Output: spam/nonspam

29 A simple machine learning algorithm K nearest neighbor (KNN) Training examples Test example Distance

30 A simple machine learning algorithm K nearest neighbor (KNN) Given a test example x, its label is predicted as the most frequent label among K training examples nearest to x. x

31 Test Protocol

32 Test protocol (1) Typical setting: Split of training/test set Apply the knowledge from training set to predict on the test set Computing the prediction accuracy or error on the test set

33 Test protocol (2) When you have parameter to tune: split of training/validation/test set Tuning the parameters of the algorithm on validation set (e.g., K in KNN) Choosing the parameters which give the best performance on validation set Training set Validation set Test set

34 Why not tuning on training set? Over fitting problem The goal of a machine learning model is to generalize to unseen data (test examples) to predict their performance Tuning on training set runs risk of fitting the model too much to training examples, trapping the model to past experience

35 Over fitting Examples extracted from T. Mitchell s slides

36 k-fold Cross-Validation k-fold cross-validation Fix the parameter to be tuned Divide dataset into k subsets of equal size Every time, use k-1 subsets to train a model Test the trained model on the remaining subset of validation examples Repeat k times Average the accuracy/error over k times experiments Validation example Training example

37 Special case: Leave one out (LOO) crossvalidation Set k=n Every time, only one example is left for validation, others are used as training examples Validation example Training example

38 Machine problem 1 Applying KNN to MNIST dataset Using three test protocols to evaluate the performance Training/test split: training examples test examples Training/validation/test split: training examples validation examples test examples 5-fold cross-validation and 10-fold cross validation (average and standard deviation) Due in two weeks (Sep 2nd before class)

39 What shall be submitted? Your implementation What language you choose to implement the algorithm? How do you search for the K nearest neighbors in the feature vector space? Exhaustive search? Or smarter one (hint: k-d tree)? Your algorithm s complexity to predict on one test example? Your results Test errors under different test protocols How performance changes when K varies? Why? Submitting your source code Electric report in PDF or Word formats Where: cap5610ucf@gmail.com.

40 Curse of Dimensionality Consider a variant nearest neighbor algorithm ε-nearest neighbor Search for all training examples within a sphere of radium ε centered at test example x in D-dimensional feature vector space Test example is predicted by majority voting among the training examples in this sphere Training examples x ε Test example

41 Curse of Dimensionality Assume all training examples are uniformly distributed in a hypercube of size r (we should have 2ε < r ), how likely does a training example fall into the ε-sphere in D-dimensional space? Volume of sphere V sphere = 2εD π D 2 DΓ D 2 Volume of hypercube V hypercube = r D x ε r The chance that an example falls into the sphere V sphere V hypercube 0, as D goes to infinity.

42 Curse of Dimensionality Curse of dimensionality is bad since no training example will fall into the sphere neighborhood as dimensionality goes to infinity. Other explanations: The distances become indiscernible as dimensionality goes big. K. Beyer, J. Goldstein, R. Ramakrishnan, U. Shaft. (1999). "When is Nearest Neighbor Meaningful?". Proc. 7th International Conference on Database Theory - ICDT'99.

43 Reading assignment Section 1.2, 1.4, Pattern Recognition and Machine Learning