Machine Learning: Methods and Applications

Transcription

1 Machine Learning: Methods and Applications Manuel Blum, Thomas Lampe, Jost Tobias Springenberg, Jan Wülfing, Prof. Dr. Martin Riedmiller Machine Learning Lab University of Freiburg October 25, 2012 Machine Learning Lab - Uni FR Proseminar WS 2012 (1)

2 Organisation - Important Numbers/Dates Block seminar at the end of the semester Preliminary dates: Topic assignment: Each topic will be worked on by groups of two Send us a list of three topics (ordered by preference) until Fr (mblum@informatik.uni-freiburg.de) List of topic assignments/tutors will be posted on the course page Once you have a topic you are committed to the seminar. Additional mandatory meeting: Make an appointment with us (2 weeks before presentation) Prepare a preliminary version of your slides Presentations: 2 20 min talk (theory + application) 10 minutes for questions Machine Learning Lab - Uni FR Proseminar WS 2012 (2)

3 Organisation - Topic assignment We have prepared 8 possible topics Each topic consists of one theory chapter from the book Machine Learning: A probabilistic perspective - K. Murphy + an application paper We want you to understand a bit of machine learning theory, but it should not be dry theory hence the applications To make it fair we will assign groups of two Each group will work on one topic together How you split the presentations is up to you Keep in mind that you wont understand the application without the theory Machine Learning Lab - Uni FR Proseminar WS 2012 (3)

4 Organisation - Presentations You have 40 min in total for your presentation You should prepare it as 2 20 min talks We want you to give an overview of the theory behind the application We want you to explain the application in depth It is okay to skip parts of the theory chapters if they are irrelevant to your application It is not okay to cut the theory explanation to 5 min just because you feel that you have not understood it ;) Think about it this way: Your explanation of the theory should stand alone and give a good overview + You should explain everything in such detail that you can explain the application with ease Machine Learning Lab - Uni FR Proseminar WS 2012 (4)

5 Topic 1: Gaussian Processes Book chapter: Application: GPPS: A Gaussian Process Positioning System for Cellular Networks - A. Schwaighofer, M. Grigoras, V. Tresp, C. Hoffmann Abstract: In this article, we present a novel approach to solving the localization problem in cellular networks. The goal is to estimate a mobile user s position, based on measurements of the signal strengths received from network base stations. Our solution works by building Gaussian process models for the distribution of signal strengths, as obtained in a series of calibration measurements. In the localization stage, the user s position can be estimated by maximizing the likelihood of received signal strengths with respect to the position. We investigate the accuracy of the proposed approach on data obtained within a large indoor cellular network. Machine Learning Lab - Uni FR Proseminar WS 2012 (5)

6 Topic 2: Support Vector Machines Book chapter: Application: Support Vector Machines for Spam Categorization - Drucker, Wu, Vapnik Abstract: We study the use of support vector machines (SVM s) in classifying as spam or nonspam by comparing it to three other classification algorithms: Ripper, Rocchio, and boosting decision trees. These four algorithms were tested on two different data sets: one data set where the number of features were constrained to the 1000 best features and another data set where the dimensionality was over SVM s performed best when using binary features. For both data sets, boosting trees and SVM s had acceptable test performance in terms of accuracy and speed. However, SVM s had significantly less training time. Machine Learning Lab - Uni FR Proseminar WS 2012 (6)

7 Topic 3: Sparse Regression Book chapter: Application: Sparse linear regression for reconstructing muscle activity from human cortical fmri - Ganesh, Burdet, Haruno, Kawato Abstract: In humans, it is generally not possible to use invasive techniques in order to identify brain activity corresponding to activity of individual muscles. Further, it is believed that the spatial resolution of non-invasive brain imaging modalities is not sufficient to isolate neural activity related to individual muscles. However, this study shows that it is possible to reconstruct muscle activity from functional magnetic resonance imaging (fmri). We simultaneously recorded surface electromyography (EMG) from two antagonist muscles and motor cortices activity using fmri, during an isometric task requiring both reciprocal activation and co-activation of the wrist muscles. Bayesian sparse regression was used to identify the parameters of a linear mapping from the fmri activity in areas 4 (M1) and 6 (pre-motor, SMA) to EMG, and to reconstruct muscle activity in an independent test data set. The mapping obtained by the sparse regression algorithm showed significantly better generalization than those obtained from algorithms commonly used in decoding, i.e., support vector machine and least square regression. The two voxel sets corresponding to the activity of the antagonist muscles were intermingled but disjoint. They were distributed over a wide area of pre-motor cortex and M1 and not limited to regions generally associated with wrist control. These results show that brain activity measured by fmri in humans can be used to predict individual muscle activity through Bayesian linear models, and that our algorithm provides a novel and non-invasive tool to investigate the brain mechanisms involved in motor control and learning in humans. Machine Learning Lab - Uni FR Proseminar WS 2012 (7)

8 Topic 4: Principal Component Analysis Book chapter: 12.2 Application: Novel Unsupervised Feature Filtering of Biological Data - Varshavsky, Gottlieb, Linial, Horn Abstract: Many methods have been developed for selecting small informative feature subsets in large noisy data. However, unsupervised methods are scarce. Examples are using the variance of data collected for each feature, or the projection of the feature on the first principal component. We propose a novel unsupervised criterion, based on SVD- entropy, selecting a feature according to its contribution to the entropy (CE) calculated on a leave-one-out basis. This can be implemented in four ways: simple ranking according to CE values (SR); forward selection by accumulating features according to which set produces highest entropy (FS1); forward selection by accumulating features through the choice of the best CE out of the remaining ones (FS2); backward elimination (BE) of features with the lowest CE. Machine Learning Lab - Uni FR Proseminar WS 2012 (8)

9 Topic 5: Hidden Markov Models Book chapter: 17 Application: Visual Recognition of American Sign Language Using Hidden Markov Models - Starner, Pentland Abstract: Hidden Markov models (HMM s) have been used prominently and successfully in speech recognition and, more recently, in handwriting recognition. Consequently, they seem ideal for visual recognition of complex, structured hand gestures such as are found in sign language. We describe an HMM-based system for recognizing sentence level American Sign Language (ASL) which attains a word accuracy of 99.2 % without explicitly modeling the fingers. Machine Learning Lab - Uni FR Proseminar WS 2012 (9)

10 Topic 6: Deep Learning Book chapter: Book Chapter 28, especially and Application: Deep Self-Taught Learning for Handwritten Character Recognition - Bengio et. al. Abstract: Recent theoretical and empirical work in statistical machine learning has demonstrated the importance of learning algorithms for deep architectures, i.e., function classes ob- tained by composing multiple non-linear transformations. Self-taught learning (exploit- ing unlabeled examples or examples from other distributions) has already been applied to deep learners, but mostly to show the advantage of unlabeled examples. Here we explore the advantage brought by out-of-distribution examples. For this purpose we developed a powerful generator of stochastic variations and noise processes for character images, in- cluding not only afne transformations but also slant, local elastic deformations, changes in thickness, background images, grey level changes, contrast, occlusion, and various types of noise. The out-of-distribution examples are obtained from these highly distorted images or by including examples of object classes different from those in the target test set. We show that deep learners benet more from them than a corresponding shallow learner, at least in the area of handwritten character recognition. In fact, we show that they reach human-level performance on both handwritten digit classication and 62-class handwritten character recognition. Machine Learning Lab - Uni FR Proseminar WS 2012 (10)

11 Topic 7: Naive Bayes Book chapter: Book Chapter 3.5 Application: A Bayesian Approach to Filtering Junk - Sahami, Dumais, Heckerman, Horvitz Abstract: In addressing the growing problem of junk on the Internet, we examine methods for the automated construction of filters to eliminate such unwanted messages from a user s mail stream. By casting this problem in a decision theoretic framework, we are able to make use of probabilistic learning methods in conjunction with a notion of differential misclassification cost to produce filters which are especially appropriate for the nuances of this task. While this may appear, at first, to be a straight-forward text classification problem, we show that by considering domain-specific features of this problem in addition to the raw text of messages, we can produce much more accurate filters. Finally, we show the efficacy of such filters in a real world usage scenario, arguing that this technology is mature enough for deployment. Machine Learning Lab - Uni FR Proseminar WS 2012 (11)

12 Topic 8: Boosting Book chapter: Book Chapter 16.4 Application: Robust Real-Time Face Detection - Viola, Jones Abstract: This paper describes a face detection framework that is capable of processing images extremely rapidly while achieving high detection rates. There are three key contributions. The first is the introduction of a new image representation called the Integral Image which allows the features used by our detector to be computed very quickly. The second is a simple and efficient classifier which is built using the AdaBoost learning algorithm (Freund and Schapire, 1995) to select a small number of critical visual features from a very large set of potential features. The third contribution is a method for combining classifiers in a cascade which allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions. A set of experiments in the domain of face detection is presented. The system yields face detection performance comparable to the best previous systems. Implemented on a conventional desktop, face detection proceeds at 15 frames per second. Machine Learning Lab - Uni FR Proseminar WS 2012 (12)