Dimension Reduction. Wei-Ta Chu 2014/10/22. Multimedia Content Analysis, CSIE, CCU
|
|
- Linette Nichols
- 8 years ago
- Views:
Transcription
1 1 Dimension Reduction Wei-Ta Chu 2014/10/22
2 2 1.1 Principal Component Analysis (PCA) Widely used in dimensionality reduction, lossy data compression, feature extraction, and data visualization Also known as Karhunen-Loeve transform Two commonly-used definitions Orthogonal projection of the data onto a lower dimensional linear space such that the variance of the projected data is maximized. Linear projection that minimizes the average projection cost C.M. Bishop, Chapter 12 of Pattern Recognition and Machine Learning, Springer, 2006.
3 Maximum Variance Formulation 3 Data set of observation {x n } with dimensionality D. Goal: project the data onto a space having dimensionality M < D with maximizing the variance of the projected data. Assume the value of M is given. Begin with M=1. Data are projected onto a line in a D-dimensional space. The direction of the line is denoted by a D-dimensional vector u 1. Each data point x n is then projected onto a scalar value u 1T x n.
4 LA Recap: Orthogonal Projection 4 proj a u = u a a 2 a (vector component of u along a) u proj a u = u u a a 2 a (vector component of u orthogonal to a) proj a u = u a a = u cosθ
5 Maximum Variance Formulation 5 The mean of the projected data is The variance of the projected data is given by Where S is the covariance matrix defined by
6 Maximum Variance Formulation 6 Maximize the projected variance with respect to u 1 Introduce a Lagrange multiplier denoted by λ 1 By setting the derivative with respect to u 1 equal to zero, we see that this quantity will have a stationary point when u 1 must be an eigenvector of S The variance will be a maximum when we set u 1 equal to the eigenvector having the largest eigenvalueλ 1
7 Maximum Variance Formulation 7 The optimal linear projection for which the variance of the projected data is maximized is now defined by the M eigenvectors u 1,, u M of the data covariance matrix S corresponding to the M largest eigenvaluesλ 1,,λ M Principal component analysis involves evaluating the mean and the covariance matrix of the data set and then finding the M eigenvectors of S corresponding the M largest eigenvalues.
8 Covariance 8 High variance, low covariance No inter-dimension dependency High variance, high covariance inter-dimension dependency
9 Minimum Error Formulation 9 Each data point can be represented by a linear combination of the basis vectors Our goal is to approximate this data point using a representation involving a restricted number M < D of variables corresponding to a projection onto a lower-dimensional subspace. M-dim projection
10 Minimum Error Formulation 10 Minimize approximation error Obtaining the minimum value of J by selecting eigenvectors to those having the D-M smallest eigenvalues, and hence the eigenvectors defining the principal subspace are those corresponding to the M largest eigenvalues. L.I. Smith, A tutorial on Principal Component Analysis, J. Shlens, A tutorial on Principal Component Analysis,
11 Applications of PCA 11 Mean vector and the first four PCA eigenvectors for the off-line digits data set Eigenvalue spectrum and the sum of the discard eigenvalues An original example together with its PCA reconstructions obtained by retaining M principal components
12 Eigenfaces 12 Eigenfaces for face recognition is a famous application of PCA Eigenfaces capture the majority of variance in face data Project a face on those eigenfaces to represent face features M. Turk and A.P. Pentland, Face recognition using eigenfaces, Proc. of CVPR, pp , 1991.
13 Singular Value Decomposition (SVD) SVD works directly on data PCA works on covariance matrix of data The SVD technique examines the entire set of data and rotates the axis to maximize variance along the first few dimensions. Problem: #1: Find concepts in text #2: Reduce dimensionality
14 SVD - Definition 14 A [n x m] = U [n x r] L [ r x r] (V [m x r] ) T A: n x m matrix (e.g., n documents, m terms) U: n x r matrix (n documents, r concepts) L: r x r diagonal matrix (strength of each concept ) (r: rank of the matrix) V: m x r matrix (m terms, r concepts)
15 SVD - Properties 15 spectral decomposition of the matrix: = x x u 1 u 2 l 1 l 2 v 1 v 2
16 SVD - Interpretation 16 documents, terms and concepts : U: document-to-concept similarity matrix V: term-to-concept similarity matrix L: its diagonal elements: strength of each concept Projection: best axis to project on: ( best = min sum of squares of projection errors)
17 SVD - Example 17 A = U L V T - example: doc-to-concept similarity matrix CS-concept MD-concept data infṛetrieval CS MD = brain lung x x
18 SVD - Example 18 A = U L V T - example: data infṛetrieval CS MD = brain lung strength of CS-concept x x
19 SVD - Example 19 CS MD A = U L V T - example: data infṛetrieval brain lung = CS-concept x term-to-concept similarity matrix x
20 SVD Dimensionality reduction 20 Q: how exactly is dim. reduction done? A: set the smallest singular values to zero: = x x
21 SVD - Dimensionality reduction ~ x 9.64 x
22 SVD - Dimensionality reduction ~
23 2.1 Multidimensional Scaling (MDS) 23 Goal: represent data points in some lowerdimensional space such that the distances between points in that space correspond to the distance between points in the original space
24 Multidimensional Scaling (MDS) 24 What MDS does is to find a set of vectors in p-dimensional space such that the matrix of Euclidean distances among them corresponds as closely as possible to some function of the input matrix according to a criterion function called stress. Stress: the degree of correspondence between the distances among points implied by MDS map and the input matrix. d ij refers to the distance between points i and j in the original space z ij refers to the distance between points i and j on the map
25 Multidimensional Scaling (MDS) 25 The true dimensionality of the data will be revealed by the rate of decline of stress as dimensionality increases.
26 Multidimensional Scaling (MDS) 26 Algorithm Assign points to arbitrary coordinates in p-dimensional space Compute Euclidean distances among all pairs of points to form a matrix Compare the matrix with the input matrix by evaluating the stress function. The smaller the value, the greater the correspondence between the two. Adjust coordinates of each point in the direction that best maximally stress Repeat steps 2 through 4 until stress won t get any lower T.F. Cox and M.A.A. Cox, Multidimensional Scaling, Chapman & Hall/CRC; 2 edition, 2000
27 Isometric Feature Mapping (Isomap) Examples J.B. Tenenbaum, V. de Silva, and J.C. Langford, A global geometric framework for nonlinear dimensionality reduction, Science, vol. 290, pp , 2000.
28 Isometric Feature Mapping (Isomap) 28 Estimate the geodesic distance between far away points, given only input-space distances. Adding up a sequence of short hops between neighboring points
29 Isometric Feature Mapping (Isomap) 29 Algorithm Step 1: construct neighborhood graph Determines which points are neighbors on the manifold Connect each point to all points within some fixed radius ε, or to its K nearest neighbors Step 2: compute shortest paths Estimate the geodesic distance between all pairs of points on the manifold by computing their shortest path in the graph Step 3: construct d-dimensional embedding Apply MDS to the matrix of graph distances constructing an embedding of the data
30 30 Isometric Feature Mapping (Isomap)
31 2.3 Locally Linear Embedding (LLE) 31 Eliminate the need to estimate pairwise distances between widely separated data points. LLE recovers global nonlinear structure from locally linear fits. S.T. Roweis and L.K. Saul, Nonlinear dimensionality reduction by locally linear embedding, Science, vol. 290, pp , oweis/lle/publications.html
32 Locally Linear Embedding (LLE) 32 Characterize the local geometry by linear coefficients that reconstruct each data point from its neighbors. Minimize the reconstruction errors Choosing d-dimensional coordinate Y i to minimize the embedding cost function
33 Example 33 The bottom images correspond to points along the top-right path, illustrating one particular mode of variability in pose and expression.
34 34 Brief Introduction of Machine Learning Techniques for Content Analysis Wei-Ta Chu 2014/10/22
35 Outline 35 Overview Gaussian Mixture Model (GMM) Hidden Markov Model (HMM) Support Vector Machine (SVM)
36 Overview 36 Any computer program that can improve its performance at some task through experience (or training) can be called a learning program. During early days, computer scientists developed learning algorithms based on heuristics and insights into human reasoning mechanisms. Decision tree, Neuro-scientists attempted to devise learning methods by imitating the structure of human brains. Y. Gong and W. Xu, Machine Learning for Multimedia Content Analysis, Springer, 2007
37 Basic Statistical Learning Problems 37 Many learning tasks can be formulated as one of the following two problems. Regression: X: input variable, Y: output variable. Infer a function f(x) so that given a value of x of the input variable X, y = f(x) is a good predication of the true value y of the output variable Y.
38 Basic Statistical Learning Problems 38 Classification: Assume that a random variable X can belong to one of a finite set of classes C={1,2,,K}. Given the value x of variable X, infer its class label l=g(x), where. It is also of great interest to estimate the probability P(k x) that X belongs to class k,. In fact both the regression and classification problems can be formulated using the same framework.
39 39 Categorizations of Machine Learning Techniques Unsupervised vs. Supervised For inferring the functions f(x) and g(x), if pairs of training data (x i,y i ) or (x i, l i ), i = 1,,N are available, then the inference process is called supervised learning. Most regression methods are supervised learning. Unsupervised methods strive to automatically partition a given data set into the predefined number of clusters also called clustering.
40 40 Categorizations of Machine Learning Techniques Generative Models vs. Discriminative Models Discriminative models strive to learn P(k x) directly from the training set without the attempt to modeling the observation x. Generative models compute P(k x) by first modeling the class-conditional probabilities P(x k) as well as the class probabilities P(k) Posterior prob. likelihood Priori prob.
41 41 Categorizations of Machine Learning Techniques Generative models: Naïve Bayes, Bayesian Networks, Gaussian Mixture Models (GMM), Hidden Markov Models (HMM), Discriminative models: Neural Networks, Support Vector Machines (SVM), Maximum Entropy Models (MEM), Conditional Random Fields (CRF),
42 42 Categorizations of Machine Learning Techniques Models for Simple Data vs. Models for Complex Data Complex data: consist of sub-entities that are strongly related one to another E.g. a beach scene usually composed of a blue sky on top, an ocean in the middle, and a sand beach at the bottom For simple: Naïve Bayes, GMM, NN, SVM For complex: BN, HMM, MEM, CRF, M 3 -net
43 43 Categorizations of Machine Learning Techniques Model Identification vs. Model Prediction Model identification: to discover an existing Law of Nature The model identification paradigm is an ill-posed problems, and is annoyed by the curse of dimensionality. The goal of model predication is to predict events well, but not necessarily through the identification of the model of events.
44 44 Gaussian Mixture Model Wei-Ta Chu 2014/10/22
45 Introduction 45 By using a sufficient number of Gaussians, and by adjusting their means and covariances as well as the coefficients in the linear combinations, almost any continuous density can be approximated to arbitrary accuracy. C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006
46 Introduction 46 Consider a superposition of K Gaussian densities Each Gaussian density is called a component of the mixture and has its own mean and covariance.
47 Introduction 47 From the sum and product rules, the marginal density is given by We can view as the prior probability of picking the kth component, and the density as the probability of x conditioned on k:
48 Introduction 48 From Baye s theorem, the posterior probability p(k x) is given by Gaussian mixture distribution is governed by parameters.. One way to set these parameters is to use maximum likelihood. likelihood Assume that different mixtures are independent and identically distributed
49 Introduction 49 In case of a single variable x, the Gaussian distribution is in the form For a D-dimensional vector x, the multivariate Gaussian distribution takes the form
50 Maximizing Likelihood 50 Setting the derivative of with respect to the means of the Gaussian components to zero The mean for the kth Gaussian component is obtained by taking a weighted mean of all of the points in the data set, in which the weighting factor for data point is given by the posterior probability
51 Maximizing Likelihood 51 Setting the derivative of with respect to the covariance of the Gaussian components to zero Each data point weighted by the corresponding posterior probability
52 Maximizing Likelihood 52 Maximize with respect to the mixing coefficients Constraint: the sum of mixing coefficients is one Using Lagrange multiplier and maximizing the following quantity If we multiply both sides by and sum over k making use of the constraint, we find. Using this to eliminate and rearranging we obtain Mixing coefficient of the kth component is given by the average responsibility which that component takes for explaining the data points
53 53 Expectation-Maximization (EM) Algorithm We first choose some initial values for the means, covariances, and mixing coefficients. Expectation step (E step) Use the current parameters to evaluate the posterior probabilities Maximization step (M step) Re-estimate the means, covariances, and mixing coefficients Each update to the parameters resulting from an E step followed by an M step is guaranteed to increase the log likelihood function.
54 Example 54
55 EM for Gaussian Mixtures 55
56 Case Study 56 Jiang, et al. A new method to segment playfield and its applications in match analysis in sports video, In Proc. of ACM MM, pp , 2004.
57 Case Study 57 The condition density of a pixel belongs to the playfield region is modeled with M Gaussian densities:
58 Related Resources 58 GMMBAYES - Bayesian Classifier and Gaussian Mixture Model ToolBox mmbayestb/ Netlab Matlab toolboxes collection
59 59 Hidden Markov Model Wei-Ta Chu 2014/10/22
60 Markov Model Chain rule 60 Assume that each of the condition distributions is independent of all previous observations except the most recent, we obtain the first-order Markov chain. First-order Markov chain Second-order Markov chain C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006
61 Example 61 What s the probability that the weather for eight consecutive days is sun-sun-sun-rain-rain-sun-cloudysun? rain sunny 3 cloudy L. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, Prentice-Hall, 1993 L. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proceedings of IEEE, vol. 77, no. 2, pp , 1989.
62 Coin-Toss Model 62 You are in a room with a curtain through which you cannot see that is happening. On the other side of the curtain is another person who is performing a coin tossing experiment (using one or more coins). The person will not tell you which coin he selects at any time; he will only tell you the result of each coin flip. A typical observation sequence would be The question is: how do we build an model to explain the observed sequence of head and tails?
63 Coin-Toss Model 63 P(H) 1-P(H) Head 1-P(H) 1 2 P(H) Tail 1-coin model (Observable Markov Model) a 11 a 22 1-a a coins model (Hidden Markov Model) P(H) = P 1 P(T) =1- P 1 P(H) = P 2 P(T) =1- P 2
64 Coin-Toss Model 64 a 11 a 22 a a 21 3-coins model (Hidden Markov Model) a 31 a 13 3 a 32 a 23 a 33 State 1 State 2 State 3 P(H) = P 1 P(H) = P 2 P(H) = P 3 P(T) =1- P 1 P(T) =1- P 2 P(T) =1- P 3
65 Elements of HMM 65 N: the number of states in the model M: the number of distinct observation symbols per state The state-transition probability A={a ij } The observation symbol probability distribution B={b j (k)} The initial state distribution To describe an HMM, we usually use the compact notation
66 Three Basic Problems of HMM 66 Problem 1: Probability Evaluation How do we compute the probability that the observed sequence was produced by the model? Scoring how well a given model matches a given observation sequence.
67 Three Basic Problems of HMM 67 Problem 2: Optimal State Sequence Attempt to uncover the hidden part of the model that is, to find the correct state sequence. For practical situations, we usually use an optimality criterion to solve this problem as best as possible.
68 Three Basic Problems of HMM 68 Problem 3: Parameter Estimation Attempt to optimize the model parameters to best describe how a given observation sequence comes about. The observation sequence used to adjust the model parameters is called a training sequence because it is used to train the HMM.
69 Solution to Problem 1 69 There are N T possible state sequences Consider one fixed-state sequence The prob. of the observation sequence given the state sequence Where we have assumed statistical independence of observations, thus we get
70 Solution to Problem 1 70 The prob. of such state sequence can be written as The joint prob. of O and q, i.e., the prob. that O and q occur simultaneously, is simply the product of the above terms The prob. of O is obtained by summing this joint prob. over all possible state sequences q, giving
71 Solution to Problem 1 71 The Forward Procedure The prob. of the partial observation sequence o 1,o 2,,o t (until time t) and state i at time t, given the model We solve for it inductively 1. Initialization 2. Induction 3. Termination
72 Solution to Problem 1 72 The Forward Procedure Require on the order of N 2 T calculations, rather than 2TN T as required by the direction calculation.
73 73 The Backward Procedure The prob. of partial observation sequence from t+1 to the end, given state i at time t and the model 1. Initialization 2. Induction
74 目 前 無 法 顯 示 此 圖 像 Solution to Problem 2 74 We define the quantity Which is the best score (highest probability) along a single path, at time t, which accounts for the first t observations and ends in state i. By induction we have
75 目 前 無 法 顯 示 此 圖 像 目 前 無 法 顯 示 此 圖 像 目 前 無 法 顯 示 此 圖 像 目 前 無 法 顯 示 此 圖 像 目 前 無 法 顯 示 此 圖 像 Solution to Problem 2 75 The Viterbi Algorithm 1. Initialization 2. Recursion 3. Termination 4. Path (state sequence) backtracking The major difference between Viterbi and the forward procedure is the maximization over previous states.
76 Solution to Problem 3 76 Choose such that its likelihood,, is locally maximized using an iterative procedure such as the Baum- Welch algorithm (also known as EM algorithm or forwardbackward algorithm) Define the prob. of being in state i at time t, and state j at time t+1, given the model and the observation sequence.
77 Solution to Problem 3 77 The prob. of being in state i at time t, given the observation sequence O and the model
78 Solution to Problem 3 78
79 Types of HMMs 79 Ergodic Left-right Parallel path left-right
80 Case Study 80 Features Field descriptor Edge descriptor Grass and sand Player height Peng, et al. Extract highlights from baseball game video with hidden Markov models, In Proc. of ICIP, vol. 1, pp , 2002.
81 Related Resources 81 Hidden Markov Model (HMM) Toolbox for Matlab html The General Hidden Markov Model library (GHMM) HTK Speech Recognition Toolkit
Manifold Learning Examples PCA, LLE and ISOMAP
Manifold Learning Examples PCA, LLE and ISOMAP Dan Ventura October 14, 28 Abstract We try to give a helpful concrete example that demonstrates how to use PCA, LLE and Isomap, attempts to provide some intuition
More informationSupervised Feature Selection & Unsupervised Dimensionality Reduction
Supervised Feature Selection & Unsupervised Dimensionality Reduction Feature Subset Selection Supervised: class labels are given Select a subset of the problem features Why? Redundant features much or
More informationText Analytics (Text Mining)
CSE 6242 / CX 4242 Apr 3, 2014 Text Analytics (Text Mining) LSI (uses SVD), Visualization Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey
More informationComparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data
CMPE 59H Comparison of Non-linear Dimensionality Reduction Techniques for Classification with Gene Expression Microarray Data Term Project Report Fatma Güney, Kübra Kalkan 1/15/2013 Keywords: Non-linear
More informationStatistical Machine Learning
Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes
More informationEM Clustering Approach for Multi-Dimensional Analysis of Big Data Set
EM Clustering Approach for Multi-Dimensional Analysis of Big Data Set Amhmed A. Bhih School of Electrical and Electronic Engineering Princy Johnson School of Electrical and Electronic Engineering Martin
More informationLinear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
More informationData, Measurements, Features
Data, Measurements, Features Middle East Technical University Dep. of Computer Engineering 2009 compiled by V. Atalay What do you think of when someone says Data? We might abstract the idea that data are
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct
More informationHardware Implementation of Probabilistic State Machine for Word Recognition
IJECT Vo l. 4, Is s u e Sp l - 5, Ju l y - Se p t 2013 ISSN : 2230-7109 (Online) ISSN : 2230-9543 (Print) Hardware Implementation of Probabilistic State Machine for Word Recognition 1 Soorya Asokan, 2
More informationNeural Networks Lesson 5 - Cluster Analysis
Neural Networks Lesson 5 - Cluster Analysis Prof. Michele Scarpiniti INFOCOM Dpt. - Sapienza University of Rome http://ispac.ing.uniroma1.it/scarpiniti/index.htm michele.scarpiniti@uniroma1.it Rome, 29
More informationMachine Learning and Data Analysis overview. Department of Cybernetics, Czech Technical University in Prague. http://ida.felk.cvut.
Machine Learning and Data Analysis overview Jiří Kléma Department of Cybernetics, Czech Technical University in Prague http://ida.felk.cvut.cz psyllabus Lecture Lecturer Content 1. J. Kléma Introduction,
More informationSocial Media Mining. Data Mining Essentials
Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers
More informationCS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #18: Dimensionality Reduc7on
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #18: Dimensionality Reduc7on Dimensionality Reduc=on Assump=on: Data lies on or near a low d- dimensional subspace Axes of this subspace
More informationPATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical
More informationClassification algorithm in Data mining: An Overview
Classification algorithm in Data mining: An Overview S.Neelamegam #1, Dr.E.Ramaraj *2 #1 M.phil Scholar, Department of Computer Science and Engineering, Alagappa University, Karaikudi. *2 Professor, Department
More informationNonlinear Iterative Partial Least Squares Method
Numerical Methods for Determining Principal Component Analysis Abstract Factors Béchu, S., Richard-Plouet, M., Fernandez, V., Walton, J., and Fairley, N. (2016) Developments in numerical treatments for
More informationCS 2750 Machine Learning. Lecture 1. Machine Learning. http://www.cs.pitt.edu/~milos/courses/cs2750/ CS 2750 Machine Learning.
Lecture Machine Learning Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square, x5 http://www.cs.pitt.edu/~milos/courses/cs75/ Administration Instructor: Milos Hauskrecht milos@cs.pitt.edu 539 Sennott
More information10-601. Machine Learning. http://www.cs.cmu.edu/afs/cs/academic/class/10601-f10/index.html
10-601 Machine Learning http://www.cs.cmu.edu/afs/cs/academic/class/10601-f10/index.html Course data All up-to-date info is on the course web page: http://www.cs.cmu.edu/afs/cs/academic/class/10601-f10/index.html
More informationProbabilistic Latent Semantic Analysis (plsa)
Probabilistic Latent Semantic Analysis (plsa) SS 2008 Bayesian Networks Multimedia Computing, Universität Augsburg Rainer.Lienhart@informatik.uni-augsburg.de www.multimedia-computing.{de,org} References
More informationVisualization by Linear Projections as Information Retrieval
Visualization by Linear Projections as Information Retrieval Jaakko Peltonen Helsinki University of Technology, Department of Information and Computer Science, P. O. Box 5400, FI-0015 TKK, Finland jaakko.peltonen@tkk.fi
More informationA Survey on Pre-processing and Post-processing Techniques in Data Mining
, pp. 99-128 http://dx.doi.org/10.14257/ijdta.2014.7.4.09 A Survey on Pre-processing and Post-processing Techniques in Data Mining Divya Tomar and Sonali Agarwal Indian Institute of Information Technology,
More information203.4770: Introduction to Machine Learning Dr. Rita Osadchy
203.4770: Introduction to Machine Learning Dr. Rita Osadchy 1 Outline 1. About the Course 2. What is Machine Learning? 3. Types of problems and Situations 4. ML Example 2 About the course Course Homepage:
More informationCourse: Model, Learning, and Inference: Lecture 5
Course: Model, Learning, and Inference: Lecture 5 Alan Yuille Department of Statistics, UCLA Los Angeles, CA 90095 yuille@stat.ucla.edu Abstract Probability distributions on structured representation.
More informationStatistical Models in Data Mining
Statistical Models in Data Mining Sargur N. Srihari University at Buffalo The State University of New York Department of Computer Science and Engineering Department of Biostatistics 1 Srihari Flood of
More informationComponent Ordering in Independent Component Analysis Based on Data Power
Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals
More informationVisualization of Large Font Databases
Visualization of Large Font Databases Martin Solli and Reiner Lenz Linköping University, Sweden ITN, Campus Norrköping, Linköping University, 60174 Norrköping, Sweden Martin.Solli@itn.liu.se, Reiner.Lenz@itn.liu.se
More informationPrincipal components analysis
CS229 Lecture notes Andrew Ng Part XI Principal components analysis In our discussion of factor analysis, we gave a way to model data x R n as approximately lying in some k-dimension subspace, where k
More informationHow To Cluster
Data Clustering Dec 2nd, 2013 Kyrylo Bessonov Talk outline Introduction to clustering Types of clustering Supervised Unsupervised Similarity measures Main clustering algorithms k-means Hierarchical Main
More informationAdvanced Signal Processing and Digital Noise Reduction
Advanced Signal Processing and Digital Noise Reduction Saeed V. Vaseghi Queen's University of Belfast UK WILEY HTEUBNER A Partnership between John Wiley & Sons and B. G. Teubner Publishers Chichester New
More informationLeast-Squares Intersection of Lines
Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a
More informationMachine Learning with MATLAB David Willingham Application Engineer
Machine Learning with MATLAB David Willingham Application Engineer 2014 The MathWorks, Inc. 1 Goals Overview of machine learning Machine learning models & techniques available in MATLAB Streamlining the
More informationReview Jeopardy. Blue vs. Orange. Review Jeopardy
Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round $200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?
More informationClass #6: Non-linear classification. ML4Bio 2012 February 17 th, 2012 Quaid Morris
Class #6: Non-linear classification ML4Bio 2012 February 17 th, 2012 Quaid Morris 1 Module #: Title of Module 2 Review Overview Linear separability Non-linear classification Linear Support Vector Machines
More informationMaster s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
More informationModelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches
Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches PhD Thesis by Payam Birjandi Director: Prof. Mihai Datcu Problematic
More information15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
More informationDATA ANALYSIS II. Matrix Algorithms
DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where
More informationSearch Taxonomy. Web Search. Search Engine Optimization. Information Retrieval
Information Retrieval INFO 4300 / CS 4300! Retrieval models Older models» Boolean retrieval» Vector Space model Probabilistic Models» BM25» Language models Web search» Learning to Rank Search Taxonomy!
More informationLecture 3: Linear methods for classification
Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,
More informationExploratory Data Analysis with MATLAB
Computer Science and Data Analysis Series Exploratory Data Analysis with MATLAB Second Edition Wendy L Martinez Angel R. Martinez Jeffrey L. Solka ( r ec) CRC Press VV J Taylor & Francis Group Boca Raton
More informationLinear Algebra Review. Vectors
Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length
More informationDistance Metric Learning in Data Mining (Part I) Fei Wang and Jimeng Sun IBM TJ Watson Research Center
Distance Metric Learning in Data Mining (Part I) Fei Wang and Jimeng Sun IBM TJ Watson Research Center 1 Outline Part I - Applications Motivation and Introduction Patient similarity application Part II
More informationA Partially Supervised Metric Multidimensional Scaling Algorithm for Textual Data Visualization
A Partially Supervised Metric Multidimensional Scaling Algorithm for Textual Data Visualization Ángela Blanco Universidad Pontificia de Salamanca ablancogo@upsa.es Spain Manuel Martín-Merino Universidad
More informationMedical Information Management & Mining. You Chen Jan,15, 2013 You.chen@vanderbilt.edu
Medical Information Management & Mining You Chen Jan,15, 2013 You.chen@vanderbilt.edu 1 Trees Building Materials Trees cannot be used to build a house directly. How can we transform trees to building materials?
More informationAn Overview of Knowledge Discovery Database and Data mining Techniques
An Overview of Knowledge Discovery Database and Data mining Techniques Priyadharsini.C 1, Dr. Antony Selvadoss Thanamani 2 M.Phil, Department of Computer Science, NGM College, Pollachi, Coimbatore, Tamilnadu,
More informationLogistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression
Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max
More informationMachine Learning and Pattern Recognition Logistic Regression
Machine Learning and Pattern Recognition Logistic Regression Course Lecturer:Amos J Storkey Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh Crichton Street,
More informationCSCI567 Machine Learning (Fall 2014)
CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /
More informationSupervised Learning (Big Data Analytics)
Supervised Learning (Big Data Analytics) Vibhav Gogate Department of Computer Science The University of Texas at Dallas Practical advice Goal of Big Data Analytics Uncover patterns in Data. Can be used
More informationVisualization of General Defined Space Data
International Journal of Computer Graphics & Animation (IJCGA) Vol.3, No.4, October 013 Visualization of General Defined Space Data John R Rankin La Trobe University, Australia Abstract A new algorithm
More informationUnsupervised and supervised dimension reduction: Algorithms and connections
Unsupervised and supervised dimension reduction: Algorithms and connections Jieping Ye Department of Computer Science and Engineering Evolutionary Functional Genomics Center The Biodesign Institute Arizona
More informationVolume 2, Issue 9, September 2014 International Journal of Advance Research in Computer Science and Management Studies
Volume 2, Issue 9, September 2014 International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online at: www.ijarcsms.com
More informationSupport Vector Machines with Clustering for Training with Very Large Datasets
Support Vector Machines with Clustering for Training with Very Large Datasets Theodoros Evgeniou Technology Management INSEAD Bd de Constance, Fontainebleau 77300, France theodoros.evgeniou@insead.fr Massimiliano
More informationIntroduction to Algorithmic Trading Strategies Lecture 2
Introduction to Algorithmic Trading Strategies Lecture 2 Hidden Markov Trading Model Haksun Li haksun.li@numericalmethod.com www.numericalmethod.com Outline Carry trade Momentum Valuation CAPM Markov chain
More informationHDDVis: An Interactive Tool for High Dimensional Data Visualization
HDDVis: An Interactive Tool for High Dimensional Data Visualization Mingyue Tan Department of Computer Science University of British Columbia mtan@cs.ubc.ca ABSTRACT Current high dimensional data visualization
More informationSection 1.1. Introduction to R n
The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to
More informationSelection of the Suitable Parameter Value for ISOMAP
1034 JOURNAL OF SOFTWARE, VOL. 6, NO. 6, JUNE 2011 Selection of the Suitable Parameter Value for ISOMAP Li Jing and Chao Shao School of Computer and Information Engineering, Henan University of Economics
More informationCS 591.03 Introduction to Data Mining Instructor: Abdullah Mueen
CS 591.03 Introduction to Data Mining Instructor: Abdullah Mueen LECTURE 3: DATA TRANSFORMATION AND DIMENSIONALITY REDUCTION Chapter 3: Data Preprocessing Data Preprocessing: An Overview Data Quality Major
More informationChristfried Webers. Canberra February June 2015
c Statistical Group and College of Engineering and Computer Science Canberra February June (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 829 c Part VIII Linear Classification 2 Logistic
More informationMachine Learning for Data Science (CS4786) Lecture 1
Machine Learning for Data Science (CS4786) Lecture 1 Tu-Th 10:10 to 11:25 AM Hollister B14 Instructors : Lillian Lee and Karthik Sridharan ROUGH DETAILS ABOUT THE COURSE Diagnostic assignment 0 is out:
More informationObject Recognition and Template Matching
Object Recognition and Template Matching Template Matching A template is a small image (sub-image) The goal is to find occurrences of this template in a larger image That is, you want to find matches of
More informationLearning is a very general term denoting the way in which agents:
What is learning? Learning is a very general term denoting the way in which agents: Acquire and organize knowledge (by building, modifying and organizing internal representations of some external reality);
More informationDimensionality Reduction - Nonlinear Methods
Chapter 3 Dimensionality Reduction - Nonlinear Methods This chapter covers various methods for nonlinear dimensionality reduction, where the nonlinear aspect refers to the mapping between the highdimensional
More informationA Survey on Outlier Detection Techniques for Credit Card Fraud Detection
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 2, Ver. VI (Mar-Apr. 2014), PP 44-48 A Survey on Outlier Detection Techniques for Credit Card Fraud
More informationSYMMETRIC EIGENFACES MILI I. SHAH
SYMMETRIC EIGENFACES MILI I. SHAH Abstract. Over the years, mathematicians and computer scientists have produced an extensive body of work in the area of facial analysis. Several facial analysis algorithms
More informationAdaptive Face Recognition System from Myanmar NRC Card
Adaptive Face Recognition System from Myanmar NRC Card Ei Phyo Wai University of Computer Studies, Yangon, Myanmar Myint Myint Sein University of Computer Studies, Yangon, Myanmar ABSTRACT Biometrics is
More informationThese slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop
Music and Machine Learning (IFT6080 Winter 08) Prof. Douglas Eck, Université de Montréal These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher
More information1 Spectral Methods for Dimensionality
1 Spectral Methods for Dimensionality Reduction Lawrence K. Saul Kilian Q. Weinberger Fei Sha Jihun Ham Daniel D. Lee How can we search for low dimensional structure in high dimensional data? If the data
More informationSubspace Analysis and Optimization for AAM Based Face Alignment
Subspace Analysis and Optimization for AAM Based Face Alignment Ming Zhao Chun Chen College of Computer Science Zhejiang University Hangzhou, 310027, P.R.China zhaoming1999@zju.edu.cn Stan Z. Li Microsoft
More informationSo which is the best?
Manifold Learning Techniques: So which is the best? Todd Wittman Math 8600: Geometric Data Analysis Instructor: Gilad Lerman Spring 2005 Note: This presentation does not contain information on LTSA, which
More informationAccurate and robust image superresolution by neural processing of local image representations
Accurate and robust image superresolution by neural processing of local image representations Carlos Miravet 1,2 and Francisco B. Rodríguez 1 1 Grupo de Neurocomputación Biológica (GNB), Escuela Politécnica
More informationIntroduction to General and Generalized Linear Models
Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby
More informationAPPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder
APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large
More informationClassification Techniques for Remote Sensing
Classification Techniques for Remote Sensing Selim Aksoy Department of Computer Engineering Bilkent University Bilkent, 06800, Ankara saksoy@cs.bilkent.edu.tr http://www.cs.bilkent.edu.tr/ saksoy/courses/cs551
More informationUnsupervised Data Mining (Clustering)
Unsupervised Data Mining (Clustering) Javier Béjar KEMLG December 01 Javier Béjar (KEMLG) Unsupervised Data Mining (Clustering) December 01 1 / 51 Introduction Clustering in KDD One of the main tasks in
More informationPrinciples of Data Mining by Hand&Mannila&Smyth
Principles of Data Mining by Hand&Mannila&Smyth Slides for Textbook Ari Visa,, Institute of Signal Processing Tampere University of Technology October 4, 2010 Data Mining: Concepts and Techniques 1 Differences
More informationA Learning Based Method for Super-Resolution of Low Resolution Images
A Learning Based Method for Super-Resolution of Low Resolution Images Emre Ugur June 1, 2004 emre.ugur@ceng.metu.edu.tr Abstract The main objective of this project is the study of a learning based method
More informationData Mining and Knowledge Discovery in Databases (KDD) State of the Art. Prof. Dr. T. Nouri Computer Science Department FHNW Switzerland
Data Mining and Knowledge Discovery in Databases (KDD) State of the Art Prof. Dr. T. Nouri Computer Science Department FHNW Switzerland 1 Conference overview 1. Overview of KDD and data mining 2. Data
More informationVideo Affective Content Recognition Based on Genetic Algorithm Combined HMM
Video Affective Content Recognition Based on Genetic Algorithm Combined HMM Kai Sun and Junqing Yu Computer College of Science & Technology, Huazhong University of Science & Technology, Wuhan 430074, China
More informationPa8ern Recogni6on. and Machine Learning. Chapter 4: Linear Models for Classifica6on
Pa8ern Recogni6on and Machine Learning Chapter 4: Linear Models for Classifica6on Represen'ng the target values for classifica'on If there are only two classes, we typically use a single real valued output
More informationMethods of Data Analysis Working with probability distributions
Methods of Data Analysis Working with probability distributions Week 4 1 Motivation One of the key problems in non-parametric data analysis is to create a good model of a generating probability distribution,
More informationNon-negative Matrix Factorization (NMF) in Semi-supervised Learning Reducing Dimension and Maintaining Meaning
Non-negative Matrix Factorization (NMF) in Semi-supervised Learning Reducing Dimension and Maintaining Meaning SAMSI 10 May 2013 Outline Introduction to NMF Applications Motivations NMF as a middle step
More informationLecture 9: Introduction to Pattern Analysis
Lecture 9: Introduction to Pattern Analysis g Features, patterns and classifiers g Components of a PR system g An example g Probability definitions g Bayes Theorem g Gaussian densities Features, patterns
More informationTrajectory based Behavior Analysis for User Verification
Trajectory based Behavior Analysis for User Verification Hsing-Kuo Pao 1, Hong-Yi Lin 1, and Kuan-Ta Chen 2 1 Dept. of Computer Science & Information Engineering, National Taiwan University of Science
More informationMultidimensional data and factorial methods
Multidimensional data and factorial methods Bidimensional data x 5 4 3 4 X 3 6 X 3 5 4 3 3 3 4 5 6 x Cartesian plane Multidimensional data n X x x x n X x x x n X m x m x m x nm Factorial plane Interpretation
More informationVisualization of Topology Representing Networks
Visualization of Topology Representing Networks Agnes Vathy-Fogarassy 1, Agnes Werner-Stark 1, Balazs Gal 1 and Janos Abonyi 2 1 University of Pannonia, Department of Mathematics and Computing, P.O.Box
More informationFeature Selection vs. Extraction
Feature Selection In many applications, we often encounter a very large number of potential features that can be used Which subset of features should be used for the best classification? Need for a small
More informationAnalysis of kiva.com Microlending Service! Hoda Eydgahi Julia Ma Andy Bardagjy December 9, 2010 MAS.622j
Analysis of kiva.com Microlending Service! Hoda Eydgahi Julia Ma Andy Bardagjy December 9, 2010 MAS.622j What is Kiva? An organization that allows people to lend small amounts of money via the Internet
More informationUW CSE Technical Report 03-06-01 Probabilistic Bilinear Models for Appearance-Based Vision
UW CSE Technical Report 03-06-01 Probabilistic Bilinear Models for Appearance-Based Vision D.B. Grimes A.P. Shon R.P.N. Rao Dept. of Computer Science and Engineering University of Washington Seattle, WA
More informationCS Master Level Courses and Areas COURSE DESCRIPTIONS. CSCI 521 Real-Time Systems. CSCI 522 High Performance Computing
CS Master Level Courses and Areas The graduate courses offered may change over time, in response to new developments in computer science and the interests of faculty and students; the list of graduate
More informationMachine Learning in Computer Vision A Tutorial. Ajay Joshi, Anoop Cherian and Ravishankar Shivalingam Dept. of Computer Science, UMN
Machine Learning in Computer Vision A Tutorial Ajay Joshi, Anoop Cherian and Ravishankar Shivalingam Dept. of Computer Science, UMN Outline Introduction Supervised Learning Unsupervised Learning Semi-Supervised
More informationHT2015: SC4 Statistical Data Mining and Machine Learning
HT2015: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Bayesian Nonparametrics Parametric vs Nonparametric
More informationIntroduction to Matrix Algebra
Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary
More informationPredict Influencers in the Social Network
Predict Influencers in the Social Network Ruishan Liu, Yang Zhao and Liuyu Zhou Email: rliu2, yzhao2, lyzhou@stanford.edu Department of Electrical Engineering, Stanford University Abstract Given two persons
More informationVirtual Landmarks for the Internet
Virtual Landmarks for the Internet Liying Tang Mark Crovella Boston University Computer Science Internet Distance Matters! Useful for configuring Content delivery networks Peer to peer applications Multiuser
More informationMachine learning for algo trading
Machine learning for algo trading An introduction for nonmathematicians Dr. Aly Kassam Overview High level introduction to machine learning A machine learning bestiary What has all this got to do with
More informationAdaptive Framework for Network Traffic Classification using Dimensionality Reduction and Clustering
IV International Congress on Ultra Modern Telecommunications and Control Systems 22 Adaptive Framework for Network Traffic Classification using Dimensionality Reduction and Clustering Antti Juvonen, Tuomo
More informationAn Introduction to Data Mining. Big Data World. Related Fields and Disciplines. What is Data Mining? 2/12/2015
An Introduction to Data Mining for Wind Power Management Spring 2015 Big Data World Every minute: Google receives over 4 million search queries Facebook users share almost 2.5 million pieces of content
More informationTensor Methods for Machine Learning, Computer Vision, and Computer Graphics
Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics Part I: Factorizations and Statistical Modeling/Inference Amnon Shashua School of Computer Science & Eng. The Hebrew University
More information