What is Machine Learning? Introduction. Example 1: Categorizing Documents. Example 1: Categorizing Documents

Transcription

1 What is Machine Learning? ntroduction Machine Learning and Pattern Recognition Chris Williams t s about finding patterns in data, and using the patterns to make predictions There are lots of problems where School of nformatics, University of Edinburgh August 2014 We d like to be able to solve them with a computer We don t know how to write a computer program to solve them We can collect examples Let s look at some example problems... (All of the slides in this course have been adapted from previous versions by Charles Sutton, Amos Storkey, David Barber.) 1 / 22 Example 1: Categorizing Documents Example 1: Categorizing Documents nput: Text of Document Label: Politics 2 / 22 Sports 3 / 22 Make a list of sport terms? What about this: Even this simple task is a bit more complicated. 4 / 22

2 Example 2: Handwriting Recognition true class = 7 true class = 2 true class = 1 true class = 0 true class = 4 true class = 1 true class = 4 true class = 9 true class = 5 Figure credit: Murphy Fig. 1.5 g high-level features using large-scale unsupervised learning ues, then picked 20 equally n. The reported accuracy racy among 20 thresholds. n in the network performs es, despite the fact that no ven during training. The achieves 81.7% accuracy in 3,026 faces in the test set, achieves 64.8%. The best ork only achieves 71% acer, selected among 100,000 om the training set, only ution, we removed the losublayers and trained the ow that the accuracy of. This agrees with preportance of local contrast., 2009). activation values for face in Figure 2. tcanbeseen, led data, theneuronlearns es and random distractors. face as an input image, the e larger than the threshold, random image as an input output value less than 0. spaced thresholds in between. The reported accuracy is the best classification accuracy among 20 thresholds Recognition Surprisingly, the best neuron in the network performs very well in recognizing faces, despite the fact that no supervisory signals were given during training. The best neuron in the network achieves 81.7% accuracy in detecting faces. There are 13,026 faces in the test set, true class = 7 true class = 2 true class = 1 so guessing all negative only achieves 64.8%. The best neuron in a one-layered network only achieves 71% accuracy while best linear filter, selected among 100,000 filters sampled randomly from the training set, only achieves 74%. true class = 0 true class = 4 true class = 1 To understand their contribution, we removed the local contrast normalization sublayers and trained the network again. Results show that the accuracy of best neuron drops to 78.5%. This agrees with pre- study class showing = 9 the true importance class = 5 of local contrast true class = 4vioustrue normalization (Jarrett et al., 2009). We visualize histograms of activation values for face images and random images in Figure 2. tcanbeseen, even with exclusively unlabeled data, theneuronlearns to differentiate between faces and random distractors. Specifically, when we give a face as an input image, the neuron tends to output value larger than the threshold, 0. n contrast, if we give a random image as an input This is deployed! All cheques and handwritten envelopes are scanned automatically image, the neuron tends to output value less than 0. Lots of other computer vision problems raise similar issues (but are harder to solve) tested neuron, by solving: x =argmin x f(x; W, H), subject to x 2 =1. Here, f(x; W, H) istheoutputofthetestedneuron given learned parameters W, H and input x. n our experiments, this constraint optimization problem is Example solved by projected 3: Find gradient descent Some withpatterns line search. These visualization methods have complementary 4.4. Visualization strengths and weaknesses. For instance, visualizing nput: 10 million images from YouTube videos the most responsive stimuli may suffer from fitting to n this section, we will present two visualization techniques to verify if the optimal stimulus of the neuron is noise. Researchers On the otherfrom hand, the Stanford numerical and optimization Google (Le et al, CML indeed 2012) approach can be susceptible to local minima. Results, a face. The first method is visualizing the most shown responsive stimuli in the test set. Since the test set Output: in FigureFind 3, confirmthatthetestedneuronindeed learns the concept of faces. something interesting. is large, this method can reliably detect near optimal stimuli of the tested neuron. The second approach is to perform numerical optimization to find the optimal stimulus (Berkes & Wiskott, 2005; Erhan et al., 2009; Le et al., 2010). n particular, we find the normbounded input x which maximizes the output f of the Building high-level features using large-scale unsupervised learning Figure 4. Scale (left) and out-of-plane (3D) rotation (right) invariance properties of the best feature. Cats! x =argmin x f(x; W, H), subject to x 2 =1. Here, f(x; W, H) istheoutputofthetestedneuron given learned parameters W, H and input x. n our experiments, this constraint optimization problem is Example solved by projected 3: Find gradient descent Some withpatterns line search. These visualization methods have complementary strengths and weaknesses. For instance, visualizing nput: 10 million images from YouTube videos the most responsive stimuli may suffer from fitting to noise. Researchers On the otherfrom hand, the Stanford numerical and optimization Google (Le et al, CML 2012) approach can be susceptible to local minima. Results, shown Output: in FigureFind 3, confirmthatthetestedneuronindeed learns the concept of something interesting. faces. Figure 3. Top: Top 48 stimuli of the best neuron from the test set. Bottom: The optimal stimulus according to numerical constraint optimization. Figure 6. Visualization of the cat face neuron (left) and human body neuron (right). Computer vision: Face detection, object recognition, scene 5 / 22 Figure 2. Histograms of faces (red) vs. no faces (blue). The test set is subsampled such that the ratio between 4.5. nvariance properties faces and no faces is one. And More Applications... We would like to assess the robustness of the face detector against common object transformations, e.g., translation, scaling and out-of-plane rotation. First, we chose a set of 10 face images and perform distortions to them, e.g., scaling and translating. For outof-plane rotation, we used 10 images of faces rotating in 3D ( out-of-plane ) understanding as the test set. To check the robustness of the neuron, we plot its averaged response over the small test set with respect to changes in scale, 3D rotation (Figure 4), and translation (Figure 5). 6 Speech processing and generation Collaborative filtering: Predict how much will like a book / movie 6 Scaled, translated faces are generated by standard cubic interpolation. For 3D rotated faces, we used 10 se- Computational advertising: Predict whether will click an ad Bioinformatics: dentify which regions of DNA encode proteins Scientific Applications: Find galaxies in images, Model cellular chemical processes Robotics: Learn a map of a building as a robot explores it Natural language processing: Syntactic parsing, building a database from text, Web search 6 / 22 s (red) vs. no faces (blue). uch that the ratio between Figure 3. Top: Top 48 stimuli of the best neuron from the test set. Bottom: The optimal stimulus according to numerical constraint optimization nvariance properties 6 / 22 For the ease of interpretation, these datasets have a positive-to-negative ratio identical to the face dataset. 7 / 22

3 Why Machine Learning? Exciting area of endeavour Data is everywhere, and growing. ML combines (some) theoretical foundations with (many) practical problems Ubiquitous in A problems (computer vision, language modelling, speech modelling, handwriting recognition) Growing demand outside of A (risk management, characterising historical artefacts, medical imaging, web analytics, recommender engines, computer games engines, financial modelling, geoinformational systems, intelligent management, operational research, etc. etc. etc.) Machine learning skills are in high demand Buzzwords: big data, analytics, data science Outline Different Types of Learning Problems The Model and the Algorithm Probabilities in Machine Learning Feature Vectors The need for assumptions/models Course Outline 8 / 22 9 / 22 Different Types of Learning Problems Supervised learning Classification Regression Unsupervised Learning Clustering Discovering latent factors many others (see Chapter 1, Murphy) Supervised Learning Given dataset D = {(x i, y i ), i = 1, 2,..., N}, learn a predictor that given a new x makes a useful statement about the associated y. Unsupervised Learning Given dataset D = {x i, i = 1, 2,..., N}, find some interesting patterns in the data set. Examples of unsupervised learning methods: Clustering Dimensionality reduction (will explain this later) Association rule learning (won t explain this; take DME) 10 / / 22

4 Principled Machine Learning The Model and the Algorithm AML gives you a toolbag of algorithms This course focuses on a principled and probabilistic view of ML What does it means to have principles (in ML)? By principles mean a theoretical framework that helps you to Understand what assumptions a learning algorithm makes Understand similarities and differences between algorithms Derive custom models and algorithms for a new learning task Model encodes understanding about the data. Process of learning from data. (e.g., a set of probability distributions p(y x)) Algorithm comes from the model, causing us to select a distribution from the set. Or multiple distributions! Different algorithms give different approximations 12 / / 22 Probabilities in Machine Learning Consider document classification again. Let x denote the document, and y the label. y { Sports, Politics } You write a function f in Java that takes x and returns y Suppose pay you 1000 for every politics article you get right, and 1M for every sports article you get right. 1 How do you modify f? Probabilities in Machine Learning Consider document classification again. Let x denote the document, and y the label. y { Sports, Politics } You write a function f in Java that takes x and returns y Suppose pay you 1000 for every politics article you get right, and 1M for every sports article you get right. 1 How do you modify f? Or to make things more complicated, suppose also charge you for every one you get wrong. Now what do you do? One answer: Don t write a function. Specify a probability distribution p(y x) Then you can make decisions by maximizing the expected profit This situation happens in real life... 1 mportant clarification: am not actually going to do this 14 / 22 1 mportant clarification: am not actually going to do this 14 / 22

5 Feature Vectors To choose ads Estimate clickthrough rate Look up what advertisers have bid Show ads with high expected value f you want to act based on your predictions, it helps to know uncertainty. We (usually) represent the input as a vector x R D. This is called a feature vector. Each element x i for i {1... D} is called a feature. Examples: Documents Let (w 1, w 2,... w V ) be a dictionary of English, e.g, w 1 = aardvark, w 2 = apple. x i the number of times that word w i appears in document (bag of words representation) mages Suppose the image is m m pixels, black and white. Let D = m 2. Order pixels from 1 to D (e.g. raster scan). Let x i {0, 1, } be the greyscale value of the pixel i 15 / / 22 The Need for Assumptions/Models y=1 y=0 x 1 x 2 Two input locations, x 1 and x 2, binary classification problem Suppose we know y(x 1 ) = 1, what does this tell us about y(x 2 )? With no assumptions it tells us nothing... A learner that makes no a priori assumptions regarding the target concept has no rational basis for classifying any unseen instances (Mitchell, 1997) Assumptions are sometimes known as inductive bias No free lunch theorem (Wolpert, 1996): there is no universally best model All learning algorithms make prior assumptions. Anyone who tells you otherwise is selling you something. 17 / / 22

6 Course Outline What is the Point of Studying this Course? Statistical Fundamentals Probability, Data and models, Bayesian methods, maximum likelihood, exponential family Supervised Learning Linear and nonlinear regression, logistic regression, neural networks Unsupervised Learning Dimensionality reduction, expectation maximization Computational ssues in Probability Distributions Optimization, Variational inference, Markov chain Monte Carlo Advanced Topics (if time) Deep learning, Gaussian processes What should you be able to do after this course? Understand why and how it is possible to do machine learning Understand how the wide set of machine learning methods fit into an overall framework Know how to use and justify these methods Be able to create your own machine learning methods Learn to think in terms of probabilistic models 19 / / 22 Summary Actions Machine learning is ubiquitous and useful Theoretical grounding helps us understand algorithms and generate new ones. No free lunch Models not algorithms Probabilistic view Attending lectures is no substitute for working through the material! Lectures will motivate the methods and approaches. Only by study of the notes and bookwork will the details be clear. f you do not understand the notes then discuss them with one another. Ask your tutors. Reading These lecture slides. Chapter 1 of Murphy. 21 / / 22