# Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data

Save this PDF as:

Size: px
Start display at page:

Download "Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data"

## Transcription

1 Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data Neil D. Lawrence Department of Computer Science, University of Sheffield, Regent Court, 211 Portobello Street, Sheffield, S1 DP, U.K. Abstract In this paper we introduce a new underlying probabilistic model for principal component analysis (PCA). Our formulation interprets PCA as a particular Gaussian process prior on a mapping from a latent space to the observed data-space. We show that if the prior s covariance function constrains the mappings to be linear the model is equivalent to PCA, we then extend the model by considering less restrictive covariance functions which allow non-linear mappings. This more general Gaussian process latent variable model (GPLVM) is then evaluated as an approach to the visualisation of high dimensional data for three different data-sets. Additionally our non-linear algorithm can be further kernelised leading to twin kernel PCA in which a mapping between feature spaces occurs. 1 Introduction Visualisation of high dimensional data can be achieved through projecting a data-set onto a lower dimensional manifold. Linear projections have traditionally been preferred due to the ease with which they can be computed. One approach to visualising a data-set in two dimensions is to project the data along two of its principal components. If we were forced to choose a priori which components to project along, we might sensibly choose those associated with the largest eigenvalues. The probabilistic reformulation of principal component analysis (PCA) also informs us that choosing the first two components is also the choice that maximises the likelihood of the data [11]. 1.1 Integrating Latent Variables, Optimising Parameters Probabilistic PCA (PPCA) is formulated as a latent variable model: given a set centred of -dimensional data and denoting the latent variable associated with each datapoint we may write the likelihood for an individual data-point under the PPCA model as!

2 + > G G! where is Gaussian distributed with unit covariance,, and. The solution for can then be found 1 by assuming that is i.i.d. and maximising the likelihood of the data-set, where T is the design matrix. Probabilistic principal component analysis and other latent variable models, such as factor analysis (FA) or independent component analysis (ICA), require a marginalisation of the latent variables and optimisation of the parameters. In this paper we consider the dual approach of marginalising and optimising each. This probabilistic model also turns out to be equivalent to PCA. 1.2 Integrating Parameters, Optimising Latent Variables By first specifying a prior distribution, where is the th row of the matrix, and integrating over we obtain a marginalised likelihood for, 35! \$#&% ('*) +, ' +.-/21 # tr, 6 T 87 (1) T where, 9:!;! < and! = T. The corresponding log-likelihood is then >? D#E% # tr, 6 T (2) Now that the parameters are marginalised we may focus on optimisation of the likelihood may be found as, with respect to the!. The gradients of (2) with respect to! H:, ; T,!,! which implies that at our solution ; T,! H! some algebraic manipulation of this formula [11] leads to T! JILKMON where ILK PRQ Q is an matrix ( is the dimension of the latent space) whose columns are eigenvectors of 6 T, M QSTQ is a diagonal matrix whose U th element is VXW Y[Z8\ ] ^ ]:_ a`, where b W is the U th eigenvalue of ; T, and N QcdQ is an arbitrary orthogonal matrix 2. Note that the eigenvalue problem we have developed can easily be shown to be equivalent to that solved in PCA (see e.g. [1]), indeed the formulation of PCA in this manner is a key step in the development of kernel PCA [9] where ; T is replaced with a kernel. Our probabilistic PCA model shares an underlying structure with [11] but differs in that where they optimise we marginalise and where they marginalise we optimise. The marginalised likelihood we are optimising in (1) is recognised as the product of independent Gaussian processes where the (linear) covariance function is given by e!6! T <. Therefore a natural extension is the non-linearisation of the mapping from latent space to the data space through the introduction of a non-linear covariance function. 1 As can the solution for f but since the solution for g is not dependent on f we will disregard it. 2 For independent component analysis the correct rotation matrix h must also be found, here we have placed no constraints on the orientation of the axes so this matrix cannot be recovered.

3 > G G, 2 Gaussian Process Latent Variable Models We saw in the previous section how PCA can be interpreted as a Gaussian process mapping 3 from a latent space to a data space where the locale of the points in latent space is determined by maximising the Gaussian process likelihood with respect to!. We will refer to models of this class as Gaussian process latent variable models (GPLVM). Principal component analysis is a GPLVM where the process prior is based on the [ inner product matrix of!, in this section we develop an alternative GPLVM by considering a prior which allows for non-linear processes, specifically we focus on the popular RBF kernel which takes the form -/ 1 Y # T _ < where is the element in the th row and th column of,, is a scale parameter and denotes the Kronecker delta. Gradients of (2) with respect to the latent points can be found through combining, 6 T, with \ via the chain rule. These gradients may be used in combination with (2) in a non-linear optimiser such as scaled conjugate gradients (SCG) [7] to obtain a latent variable representation of the data. Furthermore gradients with respect to the parameters of the kernel matrix may be computed and used to jointly optimise!,, and. The solution for! will naturally not be unique; even for the linear case described above the solution is subject to an arbitrary rotation, here we may expect multiple local minima. 2.1 Illustration of GPLVM via SCG To illustrate a simple Gaussian process latent variable model we turn to the multi-phase oil flow data [2]. This is a twelve dimensional data-set containing data of three known classes corresponding to the phase of flow in an oil pipeline: stratified, annular and homogeneous. In this illustration, for computational reasons, the data is sub-sampled to 1 data-points. Figure 1 shows visualisations of the data using both PCA and our GPLVM algorithm which required 766 iterations of SCG. The! positions for the GPLVM model were initialised using PCA (see for the MATLAB code used). The gradient based optimisation of the RBF based GPLVM s latent space shows results which are clearly superior (in terms of greater separation between the different flow domains) to those achieved by the linear PCA model. Additionally the use of a Gaussian process to perform our mapping means that there is uncertainty in the positions of the points in the data space. For our formulation the level of uncertainty is shared across all dimensions and thus may be visualised in the latent space. In Figure 1 (and subsequently) this is done through varying the intensity of the background pixels. Unfortunately, a quick analysis of the complexity of D the algorithm shows that each gradient step requires an inverse of the kernel matrix, an operation, rendering the algorithm impractical for many data-sets of interest. 3 Strictly speaking the model does not represent a mapping as a Gaussian process maps to a distribution in data space rather than a point. This apparent weakness in the model may be easily rectified to allow different levels of uncertainty for each output dimension, our more constrained model allows us to visualise this uncertainty in the latent space and is therefore preferred for this work.,

4 ! Figure 1: Visualisation of the Oil data with (a) PCA (a linear GPLVM) and (b) A GPLVM which uses an RBF kernel. Crosses, circles and plus signs represent stratified, annular and homogeneous flows respectively. The greyscales in plot (b) indicate the precision with which the manifold was expressed in data-space for that latent point. The optimised parameters of the kernel were, and f. 2.2 A Practical Algorithm for GPLVMs There are three main components to our revised, computationally efficient, optimisation process: Sparsification. Kernel methods may be sped up through sparsification, i.e. representing the data-set by a subset,, of! points known as the active set. The remainder, the inactive set, is denoted by. We make use of the informative vector machine [6] which selects points sequentially according to the reduction in the posterior process s entropy that they induce. Latent Variable Optimisation. A point from the inactive set, U, can be shown to project into the data space as a Gaussian distribution W W W W W (3) whose mean is W T, W where, denotes the kernel matrix developed from the active set and W is a column vector consisting of the elements from the U th column of, that correspond to the active set. The variance is W W W T W, W Note that since W does not appear in the inverse, gradients with respect to W do not depend on other data in. We can therefore independently optimise the likelihood of each W with respect to each W. Thus the full set! can be optimised with one pass through the data. Kernel Optimisation. The likelihood of the active set is given by #&% ' +, + ` -/21 3 # T, 7 () which can be optimised 5 with respect to, and with gradient evaluations costing Algorithm 1 summarises the order in which we implemented these steps. Note that whilst we never optimise points in the active set, we repeatedly reselect the active set so it is 5 In practice we looked for MAP solutions for all our optimisations, specifying a unit covariance Gaussian prior for the matrix and using, f and for, f and respectively.

5 Algorithm 1 An algorithm for modelling with a GPLVM. Require: A size for the active set,!. A number of iterations,. Initialise! through PCA. for iterations do Select a new active set using the IVM algorithm. Optimise () with respect to the parameters of, using scaled conjugate gradients. Select a new active set. for Each point not in active set, U. do Optimise (3) with respect to W using scaled conjugate gradients. end for end for unlikely that many points remain in their original location. For all the experiments that follow we used iterations and an active set of size!. The experiments were run on a one-shot basis 6 so we cannot make statements as to the effects that significant modification of these parameters would have. We present results on three data-sets: for the oil flow data (Figure 2) from the previous section we now make use of all 1 available points and we include a comparison with the generative topographic mapping (GTM) [] Figure 2: The full oil flow data-set visualised with (a) GTM with 225 latent points laid out on a grid and with 16 RBF nodes and (b) an RBF based GPLVM. The parameters of the latent variable model were found to be, f and. Notice how the GTM artificially discretises the latent space around the locations of the 225 latent points. We follow [5] in our 2-D visualisation of a sub-set of 3 of the digits - (6 of each digit) from a greyscale version of the USPS digit data-set (Figure 3). Finally we modelled a face data-set [8] consisting of 1965 images from a video sequence digitised at # #. Since the images are originally from a video sequence we might expect the underlying dimensionality of the data to be one the images are produced in a smooth way over time which can be thought of as a piece of string embedded in a high (56) dimensional pixel space. We therefore present ordered results from a 1-D visualisation in Figure. All the code used for performing the experiments is available from 6 By one-shot we mean that, given the algorithm above, each experiment was only run once with one setting of the random seed and the values of and given. If we were producing a visualisation for only one dataset this would leave us open to the criticism that our one-shot result was lucky. However we present three data-sets in what follows and using a one-shot approach in problems with multiple local minima removes the temptation of preferentially selecting prettier results.

6 Figure 3: The digit images visualised in the 2-D latent space. We followed [5] in plotting images in a random order but not plotting any image which would overlap an existing image. 538 of the 3 digits are plotted. Note how little space is taken by the ones (the thin line running from (-, -1.5) to (-1, )) in our visualisation, this may be contrasted with the visualisation of a similar data-set in [5]. We suggest this is because ones are easier to model and therefore do not require a large region in latent space. shef.ac.uk/~neil/gplvm/ along with avi video files of the 1-D visualisation and results from two further experiments on the same data (a 1-D GPLVM model of the digits and a 2-D GPLVM model of the faces). 3 Discussion Empirically the RBF based GPLVM model gives useful visualisations of a range of datasets. Strengths of the method include the ability to optimise the kernel parameters and to generate fantasy data from any point in latent space. Through the use of a probabilistic process we can obtain error bars on the position of the manifolds which can be visualised by imposing a greyscale image upon the latent space. When Kernels Collide: Twin Kernel PCA The eigenvalue problem which provides the maxima of (2) with respect to! for the linear kernel is exploited in kernel PCA. One could consider a twin kernel PCA where both e!6! T < and 6 T are replaced by kernel functions. Twin kernel PCA could no longer be undertaken with an eigenvalue decomposition but Algorithm 1 would still be a suitable mechanism with which to determine the values of! and the parameters of! s kernel.

7 , > > Figure : Top: Fantasy faces from the 1-D model for the face data. These faces were created by taking 6 uniformly spaced and ordered points from the latent space and visualising the mean of their distribution in data space. The plots above show this sequence unfolding (starting at the top left and moving right). Ideally the transition between the images should be smooth. Bottom: Examples from the data-set which are closest to the corresponding fantasy images in latent space. Full sequences of 2 fantasies and the entire dataset are available on the web as avi files. Stochastic neighbor embedding. Consider that (2) could be written where we have introduced a vector,, of length, 6 T and we have redefined,. The entropy of, is constant 7 in!, we therefore may add it to KL as, to (5) which is recognised Kullback-Leibler (KL) divergence between the two distributions. Stochastic neighbor embedding (SNE) [5] also minimises this KL divergence to visualise data. However in SNE the vector is discrete. Generative topographic mapping. The Generative topographic mapping [3] makes use of a radial basis function network to perform the mapping from latent space to observed space. Marginalisation of the latent space is achieved with an expectation-maximisation 7 Computing the entropy requires to be of full rank, this is not true in general but can be forced by adding jitter to, e.g..

8 (EM) algorithm. A radial basis function network is a special case of a generalised linear model and can be interpreted as a Gaussian process. Under this interpretation the GTM becomes GPLVM with a particular covariance function. The special feature of the GTM is the manner in which the latent space is represented, as a set of uniformly spaced delta functions. One could view the GPLVM as having a delta function associated with each data-point: in the GPLVM the positions of the delta functions are optimised, in the GTM each data point is associated with several different fixed delta functions. Conclusions We have presented a new class of models for probabilistic modelling and visualisation of high dimensional data. We provided strong theoretical grounding for the approach by proving that principal component analysis is a special case. On three real world data-sets we showed that visualisations provided by the model cluster the data in a reasonable way. Our model has an advantage over the various spectral clustering algorithms that have been presented in recent years in that, in common with the GTM, it is truly generative with an underlying probabilistic interpretation. However it does not suffer from the artificial discretetisation suffered by the GTM. Our theoretical analysis also suggested a novel non-linearisation of PCA involving two kernel functions. Acknowledgements We thank Aaron Hertzmann for comments on the manuscript. References [1] S. Becker, S. Thrun, and K. Obermayer, editors. Advances in Neural Information Processing Systems, volume 15, Cambridge, MA, 23. MIT Press. [2] C. M. Bishop and G. D. James. Analysis of multiphase flows using dual-energy gamma densitometry and neural networks. Nuclear Instruments and Methods in Physics Research, A327:58 593, [3] C. M. Bishop, M. Svensén, and C. K. I. Williams. GTM: a principled alternative to the Self- Organizing Map. In Advances in Neural Information Processing Systems, volume 9, pages MIT Press, [] C. M. Bishop, M. Svensén, and C. K. I. Williams. GTM: the Generative Topographic Mapping. Neural Computation, 1(1):215 23, [5] G. Hinton and S. Roweis. Stochastic neighbor embedding. In Becker et al. [1], pages [6] N. D. Lawrence, M. Seeger, and R. Herbrich. Fast sparse Gaussian process methods: The informative vector machine. In Becker et al. [1], pages [7] I. T. Nabney. Netlab: Algorithms for Pattern Recognition. Advances in Pattern Recognition. Springer, Berlin, 21. Code available from [8] S. Roweis, L. K. Saul, and G. Hinton. Global coordination of local linear models. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems, volume 1, pages , Cambridge, MA, 22. MIT Press. [9] B. Schölkopf, A. J. Smola, and K.-R. Müller. Kernel principal component analysis. In Proceedings 1997 International Conference on Artificial Neural Networks, ICANN 97, page 583, Lausanne, Switzerland, [1] M. E. Tipping. Sparse kernel principal component analysis. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13, pages , Cambridge, MA, 21. MIT Press. [11] M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, B, 6(3): , 1999.

### Exploratory Data Analysis Using Radial Basis Function Latent Variable Models

Exploratory Data Analysis Using Radial Basis Function Latent Variable Models Alan D. Marrs and Andrew R. Webb DERA St Andrews Road, Malvern Worcestershire U.K. WR14 3PS {marrs,webb}@signal.dera.gov.uk

### Component Ordering in Independent Component Analysis Based on Data Power

Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals

### Learning Shared Latent Structure for Image Synthesis and Robotic Imitation

University of Washington CSE TR 2005-06-02 To appear in Advances in NIPS 18, 2006 Learning Shared Latent Structure for Image Synthesis and Robotic Imitation Aaron P. Shon Keith Grochow Aaron Hertzmann

### Probabilistic Visualisation of High-dimensional Binary Data

Probabilistic Visualisation of High-dimensional Binary Data Michael E. Tipping Microsoft Research, St George House, 1 Guildhall Street, Cambridge CB2 3NH, U.K. mtipping@microsoit.com Abstract We present

### arxiv:1410.4984v1 [cs.dc] 18 Oct 2014

Gaussian Process Models with Parallelization and GPU acceleration arxiv:1410.4984v1 [cs.dc] 18 Oct 2014 Zhenwen Dai Andreas Damianou James Hensman Neil Lawrence Department of Computer Science University

### Machine Learning and Pattern Recognition Logistic Regression

Machine Learning and Pattern Recognition Logistic Regression Course Lecturer:Amos J Storkey Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh Crichton Street,

### Cheng Soon Ong & Christfried Webers. Canberra February June 2016

c Cheng Soon Ong & Christfried Webers Research Group and College of Engineering and Computer Science Canberra February June (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 31 c Part I

### Bayesian Statistics: Indian Buffet Process

Bayesian Statistics: Indian Buffet Process Ilker Yildirim Department of Brain and Cognitive Sciences University of Rochester Rochester, NY 14627 August 2012 Reference: Most of the material in this note

### Triangular Visualization

Triangular Visualization Tomasz Maszczyk and W lodzis law Duch Department of Informatics, Nicolaus Copernicus University, Toruń, Poland tmaszczyk@is.umk.pl;google:w.duch http://www.is.umk.pl Abstract.

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Visualization by Linear Projections as Information Retrieval

Visualization by Linear Projections as Information Retrieval Jaakko Peltonen Helsinki University of Technology, Department of Information and Computer Science, P. O. Box 5400, FI-0015 TKK, Finland jaakko.peltonen@tkk.fi

### Bayesian Image Super-Resolution

Bayesian Image Super-Resolution Michael E. Tipping and Christopher M. Bishop Microsoft Research, Cambridge, U.K..................................................................... Published as: Bayesian

### Soft Clustering with Projections: PCA, ICA, and Laplacian

1 Soft Clustering with Projections: PCA, ICA, and Laplacian David Gleich and Leonid Zhukov Abstract In this paper we present a comparison of three projection methods that use the eigenvectors of a matrix

### Lecture 9: Continuous

CSC2515 Fall 2007 Introduction to Machine Learning Lecture 9: Continuous Latent Variable Models 1 Example: continuous underlying variables What are the intrinsic latent dimensions in these two datasets?

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics

Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics Part I: Factorizations and Statistical Modeling/Inference Amnon Shashua School of Computer Science & Eng. The Hebrew University

### Learning Gaussian process models from big data. Alan Qi Purdue University Joint work with Z. Xu, F. Yan, B. Dai, and Y. Zhu

Learning Gaussian process models from big data Alan Qi Purdue University Joint work with Z. Xu, F. Yan, B. Dai, and Y. Zhu Machine learning seminar at University of Cambridge, July 4 2012 Data A lot of

### Modelling Handwitten Digit Data using Probabilistic Principal Component Analysis

Modelling Handwitten Digit Data using Probabilistic Principal Component Analysis Mohamed E.M. Musa, Robert P.W. Duin and Dick de Ridder Pattern Recognition Group, Department of Applied Physics Delft University

### Accurate and robust image superresolution by neural processing of local image representations

Accurate and robust image superresolution by neural processing of local image representations Carlos Miravet 1,2 and Francisco B. Rodríguez 1 1 Grupo de Neurocomputación Biológica (GNB), Escuela Politécnica

### Supporting Online Material for

www.sciencemag.org/cgi/content/full/313/5786/504/dc1 Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks G. E. Hinton* and R. R. Salakhutdinov *To whom correspondence

### Manifold Learning Examples PCA, LLE and ISOMAP

Manifold Learning Examples PCA, LLE and ISOMAP Dan Ventura October 14, 28 Abstract We try to give a helpful concrete example that demonstrates how to use PCA, LLE and Isomap, attempts to provide some intuition

### Nonlinear Iterative Partial Least Squares Method

Numerical Methods for Determining Principal Component Analysis Abstract Factors Béchu, S., Richard-Plouet, M., Fernandez, V., Walton, J., and Fairley, N. (2016) Developments in numerical treatments for

### Gaussian Processes in Machine Learning

Gaussian Processes in Machine Learning Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics, 72076 Tübingen, Germany carl@tuebingen.mpg.de WWW home page: http://www.tuebingen.mpg.de/ carl

### Filtered Gaussian Processes for Learning with Large Data-Sets

Filtered Gaussian Processes for Learning with Large Data-Sets Jian Qing Shi, Roderick Murray-Smith 2,3, D. Mike Titterington 4,and Barak A. Pearlmutter 3 School of Mathematics and Statistics, University

### 1 Spectral Methods for Dimensionality

1 Spectral Methods for Dimensionality Reduction Lawrence K. Saul Kilian Q. Weinberger Fei Sha Jihun Ham Daniel D. Lee How can we search for low dimensional structure in high dimensional data? If the data

### IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 7, JULY 2009 1181

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 7, JULY 2009 1181 The Global Kernel k-means Algorithm for Clustering in Feature Space Grigorios F. Tzortzis and Aristidis C. Likas, Senior Member, IEEE

### Spectral Methods for Dimensionality Reduction

Spectral Methods for Dimensionality Reduction Prof. Lawrence Saul Dept of Computer & Information Science University of Pennsylvania UCLA IPAM Tutorial, July 11-14, 2005 Dimensionality reduction Question

### Comparing GPLVM Approaches for Dimensionality Reduction in Character Animation

Comparing GPLVM Approaches for Dimensionality Reduction in Character Animation Sébastien Quirion 1 Chantale Duchesne 1 Denis Laurendeau 1 Mario Marchand 2 1 Computer Vision and Systems Laboratory Dept.

### Mixtures of Robust Probabilistic Principal Component Analyzers

Mixtures of Robust Probabilistic Principal Component Analyzers Cédric Archambeau, Nicolas Delannay 2 and Michel Verleysen 2 - University College London, Dept. of Computer Science Gower Street, London WCE

### Principal Component Analysis Application to images

Principal Component Analysis Application to images Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering, Department of Cybernetics Center for Machine Perception http://cmp.felk.cvut.cz/

### LABEL PROPAGATION ON GRAPHS. SEMI-SUPERVISED LEARNING. ----Changsheng Liu 10-30-2014

LABEL PROPAGATION ON GRAPHS. SEMI-SUPERVISED LEARNING ----Changsheng Liu 10-30-2014 Agenda Semi Supervised Learning Topics in Semi Supervised Learning Label Propagation Local and global consistency Graph

### Least-Squares Intersection of Lines

Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a

### EM Clustering Approach for Multi-Dimensional Analysis of Big Data Set

EM Clustering Approach for Multi-Dimensional Analysis of Big Data Set Amhmed A. Bhih School of Electrical and Electronic Engineering Princy Johnson School of Electrical and Electronic Engineering Martin

### Introduction to Support Vector Machines. Colin Campbell, Bristol University

Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.

### Face Recognition using Principle Component Analysis

Face Recognition using Principle Component Analysis Kyungnam Kim Department of Computer Science University of Maryland, College Park MD 20742, USA Summary This is the summary of the basic idea about PCA

### A Introduction to Matrix Algebra and Principal Components Analysis

A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2 Today s Class An introduction to matrix algebra

### Support Vector Machines with Clustering for Training with Very Large Datasets

Support Vector Machines with Clustering for Training with Very Large Datasets Theodoros Evgeniou Technology Management INSEAD Bd de Constance, Fontainebleau 77300, France theodoros.evgeniou@insead.fr Massimiliano

### CSE 494 CSE/CBS 598 (Fall 2007): Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye

CSE 494 CSE/CBS 598 Fall 2007: Numerical Linear Algebra for Data Exploration Clustering Instructor: Jieping Ye 1 Introduction One important method for data compression and classification is to organize

### Supervised Feature Selection & Unsupervised Dimensionality Reduction

Supervised Feature Selection & Unsupervised Dimensionality Reduction Feature Subset Selection Supervised: class labels are given Select a subset of the problem features Why? Redundant features much or

### Scaling Kernel-Based Systems to Large Data Sets

Data Mining and Knowledge Discovery, Volume 5, Number 3, 200 Scaling Kernel-Based Systems to Large Data Sets Volker Tresp Siemens AG, Corporate Technology Otto-Hahn-Ring 6, 8730 München, Germany (volker.tresp@mchp.siemens.de)

### Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms

Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms Scott Pion and Lutz Hamel Abstract This paper presents the results of a series of analyses performed on direct mail

### Automated Hierarchical Mixtures of Probabilistic Principal Component Analyzers

Automated Hierarchical Mixtures of Probabilistic Principal Component Analyzers Ting Su tsu@ece.neu.edu Jennifer G. Dy jdy@ece.neu.edu Department of Electrical and Computer Engineering, Northeastern University,

### Machine Learning and Data Analysis overview. Department of Cybernetics, Czech Technical University in Prague. http://ida.felk.cvut.

Machine Learning and Data Analysis overview Jiří Kléma Department of Cybernetics, Czech Technical University in Prague http://ida.felk.cvut.cz psyllabus Lecture Lecturer Content 1. J. Kléma Introduction,

### Learning to Find Pre-Images

Learning to Find Pre-Images Gökhan H. Bakır, Jason Weston and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics Spemannstraße 38, 72076 Tübingen, Germany {gb,weston,bs}@tuebingen.mpg.de

### Face Recognition using SIFT Features

Face Recognition using SIFT Features Mohamed Aly CNS186 Term Project Winter 2006 Abstract Face recognition has many important practical applications, like surveillance and access control.

### Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014

Probabilistic Models for Big Data Alex Davies and Roger Frigola University of Cambridge 13th February 2014 The State of Big Data Why probabilistic models for Big Data? 1. If you don t have to worry about

### CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES. From Exploratory Factor Analysis Ledyard R Tucker and Robert C.

CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES From Exploratory Factor Analysis Ledyard R Tucker and Robert C MacCallum 1997 180 CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES In

### Flexible and efficient Gaussian process models for machine learning

Flexible and efficient Gaussian process models for machine learning Edward Lloyd Snelson M.A., M.Sci., Physics, University of Cambridge, UK (2001) Gatsby Computational Neuroscience Unit University College

### A Sampled Texture Prior for Image Super-Resolution

A Sampled Texture Prior for Image Super-Resolution Lyndsey C. Pickup, Stephen J. Roberts and Andrew Zisserman Robotics Research Group Department of Engineering Science University of Oxford Parks Road,

### Linear Algebra. Introduction

Linear Algebra Caren Diefenderfer, Hollins University, chair David Hill, Temple University, lead writer Sheldon Axler, San Francisco State University Nancy Neudauer, Pacific University David Strong, Pepperdine

### Ranking on Data Manifolds

Ranking on Data Manifolds Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics, 72076 Tuebingen, Germany {firstname.secondname

### Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data

CMPE 59H Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data Term Project Report Fatma Güney, Kübra Kalkan 1/15/2013 Keywords: Non-linear

### Introduction to Machine Learning. Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011

Introduction to Machine Learning Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011 1 Outline 1. What is machine learning? 2. The basic of machine learning 3. Principles and effects of machine learning

### A Learning Based Method for Super-Resolution of Low Resolution Images

A Learning Based Method for Super-Resolution of Low Resolution Images Emre Ugur June 1, 2004 emre.ugur@ceng.metu.edu.tr Abstract The main objective of this project is the study of a learning based method

### Priors for People Tracking from Small Training Sets

Priors for People Tracking from Small Training Sets Raquel Urtasun CVLab EPFL, Lausanne Switzerland David J. Fleet Computer Science Dept. University of Toronto Canada Aaron Hertzmann Computer Science Dept.

### Dimension Reduction. Wei-Ta Chu 2014/10/22. Multimedia Content Analysis, CSIE, CCU

1 Dimension Reduction Wei-Ta Chu 2014/10/22 2 1.1 Principal Component Analysis (PCA) Widely used in dimensionality reduction, lossy data compression, feature extraction, and data visualization Also known

1 1 Introduction Lecturer: Prof. Aude Billard (aude.billard@epfl.ch) Teaching Assistants: Guillaume de Chambrier, Nadia Figueroa, Denys Lamotte, Nicola Sommer 2 2 Course Format Alternate between: Lectures

### Self Organizing Maps: Fundamentals

Self Organizing Maps: Fundamentals Introduction to Neural Networks : Lecture 16 John A. Bullinaria, 2004 1. What is a Self Organizing Map? 2. Topographic Maps 3. Setting up a Self Organizing Map 4. Kohonen

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### Lecture Topic: Low-Rank Approximations

Lecture Topic: Low-Rank Approximations Low-Rank Approximations We have seen principal component analysis. The extraction of the first principle eigenvalue could be seen as an approximation of the original

### Predict Influencers in the Social Network

Predict Influencers in the Social Network Ruishan Liu, Yang Zhao and Liuyu Zhou Email: rliu2, yzhao2, lyzhou@stanford.edu Department of Electrical Engineering, Stanford University Abstract Given two persons

### UW CSE Technical Report 03-06-01 Probabilistic Bilinear Models for Appearance-Based Vision

UW CSE Technical Report 03-06-01 Probabilistic Bilinear Models for Appearance-Based Vision D.B. Grimes A.P. Shon R.P.N. Rao Dept. of Computer Science and Engineering University of Washington Seattle, WA

### Lecture 11: Graphical Models for Inference

Lecture 11: Graphical Models for Inference So far we have seen two graphical models that are used for inference - the Bayesian network and the Join tree. These two both represent the same joint probability

### Maximum Likelihood Graph Structure Estimation with Degree Distributions

Maximum Likelihood Graph Structure Estimation with Distributions Bert Huang Computer Science Department Columbia University New York, NY 17 bert@cs.columbia.edu Tony Jebara Computer Science Department

### Principal components analysis

CS229 Lecture notes Andrew Ng Part XI Principal components analysis In our discussion of factor analysis, we gave a way to model data x R n as approximately lying in some k-dimension subspace, where k

### A Semi-parametric Approach for Decomposition of Absorption Spectra in the Presence of Unknown Components

A Semi-parametric Approach for Decomposition of Absorption Spectra in the Presence of Unknown Components Payman Sadegh 1,2, Henrik Aalborg Nielsen 1, and Henrik Madsen 1 Abstract Decomposition of absorption

### The Second Undergraduate Level Course in Linear Algebra

The Second Undergraduate Level Course in Linear Algebra University of Massachusetts Dartmouth Joint Mathematics Meetings New Orleans, January 7, 11 Outline of Talk Linear Algebra Curriculum Study Group

### Generalized Neutral Portfolio

Generalized Neutral Portfolio Sandeep Bhutani bhutanister@gmail.com December 10, 2009 Abstract An asset allocation framework decomposes the universe of asset returns into factors either by Fundamental

### ViSOM A Novel Method for Multivariate Data Projection and Structure Visualization

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 1, JANUARY 2002 237 ViSOM A Novel Method for Multivariate Data Projection and Structure Visualization Hujun Yin Abstract When used for visualization of

### Factor analysis. Angela Montanari

Factor analysis Angela Montanari 1 Introduction Factor analysis is a statistical model that allows to explain the correlations between a large number of observed correlated variables through a small number

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Information Retrieval Perspective to Interactive Data Visualization

Eurographics Conference on Visualization (EuroVis) (2013) M. Hlawitschka and T. Weinkauf (Editors) Short Papers Information Retrieval Perspective to Interactive Data Visualization J. Peltonen 1, M. Sandholm

### Real-time Body Tracking Using a Gaussian Process Latent Variable Model

Real-time Body Tracking Using a Gaussian Process Latent Variable Model Shaobo Hou Aphrodite Galata Fabrice Caillette Neil Thacker Paul Bromiley School of Computer Science The University of Manchester ISBE

### Chapter 7. Diagnosis and Prognosis of Breast Cancer using Histopathological Data

Chapter 7 Diagnosis and Prognosis of Breast Cancer using Histopathological Data In the previous chapter, a method for classification of mammograms using wavelet analysis and adaptive neuro-fuzzy inference

### P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition

P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition K. Osypov* (WesternGeco), D. Nichols (WesternGeco), M. Woodward (WesternGeco) & C.E. Yarman (WesternGeco) SUMMARY Tomographic

### Learning with Local and Global Consistency

Learning with Local and Global Consistency Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics, 7276 Tuebingen, Germany

### Using Mixtures-of-Distributions models to inform farm size selection decisions in representative farm modelling. Philip Kostov and Seamus McErlean

Using Mixtures-of-Distributions models to inform farm size selection decisions in representative farm modelling. by Philip Kostov and Seamus McErlean Working Paper, Agricultural and Food Economics, Queen

### Supervised Learning with Unsupervised Output Separation

Supervised Learning with Unsupervised Output Separation Nathalie Japkowicz School of Information Technology and Engineering University of Ottawa 150 Louis Pasteur, P.O. Box 450 Stn. A Ottawa, Ontario,

### Learning with Local and Global Consistency

Learning with Local and Global Consistency Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf Max Planck Institute for Biological Cybernetics, 7276 Tuebingen, Germany

### Intelligent Data Analysis. Topographic Maps of Vectorial Data. School of Computer Science University of Birmingham

Intelligent Data Analysis Topographic Maps of Vectorial Data Peter Tiňo School of Computer Science University of Birmingham Discovering low-dimensional spatial layout in higher dimensional spaces - 1-D/3-D

### Computing with Finite and Infinite Networks

Computing with Finite and Infinite Networks Ole Winther Theoretical Physics, Lund University Sölvegatan 14 A, S-223 62 Lund, Sweden winther@nimis.thep.lu.se Abstract Using statistical mechanics results,

### Discovering Structure in Continuous Variables Using Bayesian Networks

In: "Advances in Neural Information Processing Systems 8," MIT Press, Cambridge MA, 1996. Discovering Structure in Continuous Variables Using Bayesian Networks Reimar Hofmann and Volker Tresp Siemens AG,

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### Statistical machine learning, high dimension and big data

Statistical machine learning, high dimension and big data S. Gaïffas 1 14 mars 2014 1 CMAP - Ecole Polytechnique Agenda for today Divide and Conquer principle for collaborative filtering Graphical modelling,

### arxiv:1402.4293v1 [stat.ml] 18 Feb 2014

The Random Forest Kernel and creating other kernels for big data from random partitions arxiv:1402.4293v1 [stat.ml] 18 Feb 2014 Alex Davies Dept of Engineering, University of Cambridge Cambridge, UK Zoubin

### Lecture 19 Camera Matrices and Calibration

Lecture 19 Camera Matrices and Calibration Project Suggestions Texture Synthesis for In-Painting Section 10.5.1 in Szeliski Text Project Suggestions Image Stitching (Chapter 9) Face Recognition Chapter

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Class #6: Non-linear classification. ML4Bio 2012 February 17 th, 2012 Quaid Morris

Class #6: Non-linear classification ML4Bio 2012 February 17 th, 2012 Quaid Morris 1 Module #: Title of Module 2 Review Overview Linear separability Non-linear classification Linear Support Vector Machines

### CNFSAT: Predictive Models, Dimensional Reduction, and Phase Transition

CNFSAT: Predictive Models, Dimensional Reduction, and Phase Transition Neil P. Slagle College of Computing Georgia Institute of Technology Atlanta, GA npslagle@gatech.edu Abstract CNFSAT embodies the P

### Nonlinear Dimensionality Reduction

Nonlinear Dimensionality Reduction Piyush Rai CS5350/6350: Machine Learning October 25, 2011 Recap: Linear Dimensionality Reduction Linear Dimensionality Reduction: Based on a linear projection of the

### Methods of Data Analysis Working with probability distributions

Methods of Data Analysis Working with probability distributions Week 4 1 Motivation One of the key problems in non-parametric data analysis is to create a good model of a generating probability distribution,

### Dimensionality Reduction A Short Tutorial

Dimensionality Reduction A Short Tutorial Ali Ghodsi Department of Statistics and Actuarial Science University of Waterloo Waterloo, Ontario, Canada, 2006 c Ali Ghodsi, 2006 Contents 1 An Introduction

### Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector

### Visualization of large data sets using MDS combined with LVQ.

Visualization of large data sets using MDS combined with LVQ. Antoine Naud and Włodzisław Duch Department of Informatics, Nicholas Copernicus University, Grudziądzka 5, 87-100 Toruń, Poland. www.phys.uni.torun.pl/kmk

### APPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder

APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large

### Tetris: Experiments with the LP Approach to Approximate DP

Tetris: Experiments with the LP Approach to Approximate DP Vivek F. Farias Electrical Engineering Stanford University Stanford, CA 94403 vivekf@stanford.edu Benjamin Van Roy Management Science and Engineering

### MUSICAL INSTRUMENT FAMILY CLASSIFICATION

MUSICAL INSTRUMENT FAMILY CLASSIFICATION Ricardo A. Garcia Media Lab, Massachusetts Institute of Technology 0 Ames Street Room E5-40, Cambridge, MA 039 USA PH: 67-53-0 FAX: 67-58-664 e-mail: rago @ media.

### Subspace Analysis and Optimization for AAM Based Face Alignment

Subspace Analysis and Optimization for AAM Based Face Alignment Ming Zhao Chun Chen College of Computer Science Zhejiang University Hangzhou, 310027, P.R.China zhaoming1999@zju.edu.cn Stan Z. Li Microsoft