# CS229 Lecture notes. Andrew Ng

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 CS229 Lecture notes Andrew Ng Part X Factor analysis Whenwehavedatax (i) R n thatcomesfromamixtureofseveral Gaussians, the EM algorithm can be applied to fit a mixture model. In this setting, we usually imagine problems where we have sufficient data to be able to discern the multiple-gaussian structure in the data. For instance, this would be the case if our training set size m was significantly larger than the dimension n of the data. Now, consider a setting in which n m. In such a problem, it might be difficulttomodelthedataevenwithasinglegaussian, muchlessamixtureof Gaussian. Specifically, since the m data points span only a low-dimensional subspace of R n, if we model the data as Gaussian, and estimate the mean and covariance using the usual maximum likelihood estimators, µ = 1 m Σ = 1 m x (i) (x (i) µ)(x (i) µ) T, we would find that the matrix Σ is singular. This means that Σ 1 does not exist, and 1/ Σ 1/2 = 1/0. But both of these terms are needed in computing the usual density of a multivariate Gaussian distribution. Another way of stating this difficulty is that maximum likelihood estimates of the parameters result in a Gaussian that places all of its probability in the affine space spanned by the data, 1 and this corresponds to a singular covariance matrix This is the set of points x satisfying x = m α ix (i), for some α i s so that m α 1 = 1

2 2 More generally, unless m exceeds n by some reasonable amount, the maximum likelihood estimates of the mean and covariance may be quite poor. Nonetheless, we would still like to be able to fit a reasonable Gaussian model to the data, and perhaps capture some interesting covariance structure in the data. How can we do this? In the next section, we begin by reviewing two possible restrictions on Σ, ones that allow us to fit Σ with small amounts of data but neither of which will give a satisfactory solution to our problem. We next discuss some properties of Gaussians that will be needed later; specifically, how to find marginal and conditonal distributions of Gaussians. Finally, we present the factor analysis model, and EM for it. 1 Restrictions of Σ If we do not have sufficient data to fit a full covariance matrix, we may place some restrictions on the space of matrices Σ that we will consider. For instance, we may choose to fit a covariance matrix Σ that is diagonal. In this setting, the reader may easily verify that the maximum likelihood estimate of the covariance matrix is given by the diagonal matrix Σ satisfying Σ jj = 1 m (x (i) j µ j ) 2. Thus, Σ jj is just the empirical estimate of the variance of the j-th coordinate of the data. Recall that the contours of a Gaussian density are ellipses. A diagonal Σ corresponds to a Gaussian where the major axes of these ellipses are axisaligned. Sometimes, we may place a further restriction on the covariance matrix that not only must it be diagonal, but its diagonal entries must all be equal. Inthissetting, wehaveσ = σ 2 I, whereσ 2 istheparameterunderourcontrol. The maximum likelihood estimate of σ 2 can be found to be: σ 2 = 1 mn n j=1 (x (i) j µ j ) 2. This model corresponds to using Gaussians whose densities have contours that are circles (in 2 dimensions; or spheres/hyperspheres in higher dimensions).

3 3 If we were fitting a full, unconstrained, covariance matrix Σ to data, it was necessary that m n+1 in order for the maximum likelihood estimate of Σ not to be singular. Under either of the two restrictions above, we may obtain non-singular Σ when m 2. However, restricting Σ to be diagonal also means modeling the different coordinates x i, x j of the data as being uncorrelated and independent. Often, it would be nice to be able to capture some interesting correlation structure in the data. If we were to use either of the restrictions on Σ described above, we would therefore fail to do so. In this set of notes, we will describe the factor analysis model, which uses more parameters than the diagonal Σ and captures some correlations in the data, but also without having to fit a full covariance matrix. 2 Marginals and conditionals of Gaussians Before describing factor analysis, we digress to talk about how to find conditional and marginal distributions of random variables with a joint multivariate Gaussian distribution. Suppose we have a vector-valued random variable [ ] x1 x =, where x 1 R r, x 2 R s, and x R r+s. Suppose x N(µ,Σ), where [ ] [ ] µ1 Σ11 Σ µ =, Σ = 12. Σ 21 Σ 22 µ 2 Here, µ 1 R r, µ 2 R s, Σ 11 R r r, Σ 12 R r s, and so on. Note that since covariance matrices are symmetric, Σ 12 = Σ T 21. Under our assumptions, x 1 and x 2 are jointly multivariate Gaussian. What isthe marginal distribution of x 1? It is not hardto see that E[x 1 ] = µ 1, and that Cov(x 1 ) = E[(x 1 µ 1 )(x 1 µ 1 )] = Σ 11. To see that the latter is true, note that by definition of the joint covariance of x 1 and x 2, we have x 2

4 4 that Cov(x) = Σ [ ] Σ11 Σ = 12 Σ 21 Σ 22 = E[(x µ)(x µ) T ] [ ( )( ) ] T x1 µ = E 1 x1 µ 1 x 2 µ 2 x 2 µ 2 [ ] (x1 µ = E 1 )(x 1 µ 1 ) T (x 1 µ 1 )(x 2 µ 2 ) T (x 2 µ 2 )(x 1 µ 1 ) T (x 2 µ 2 )(x 2 µ 2 ) T. Matching the upper-left subblocks in the matrices in the second and the last lines above gives the result. Since marginal distributions of Gaussians are themselves Gaussian, we thereforehavethatthemarginaldistributionofx 1 isgivenbyx 1 N(µ 1,Σ 11 ). Also, we can ask, what is the conditional distribution of x 1 given x 2? By referring to the definition of the multivariate Gaussian distribution, it can be shown that x 1 x 2 N(µ 1 2,Σ 1 2 ), where µ 1 2 = µ 1 +Σ 12 Σ 1 22 (x 2 µ 2 ), (1) Σ 1 2 = Σ 11 Σ 12 Σ 1 22Σ 21. (2) When working with the factor analysis model in the next section, these formulas for finding conditional and marginal distributions of Gaussians will be very useful. 3 The Factor analysis model In the factor analysis model, we posit a joint distribution on (x,z) as follows, where z R k is a latent random variable: z N(0,I) x z N(µ+Λz,Ψ). Here, the parameters of our model are the vector µ R n, the matrix Λ R n k, and the diagonal matrix Ψ R n n. The value of k is usually chosen to be smaller than n.

5 5 Thus, we imagine that each datapoint x (i) is generated by sampling a k dimension multivariate Gaussian z (i). Then, it is mapped to a k-dimensional affine space of R n by computing µ+λz (i). Lastly, x (i) is generated by adding covariance Ψ noise to µ+λz (i). Equivalently (convince yourself that this is the case), we can therefore also define the factor analysis model according to z N(0,I) ǫ N(0,Ψ) x = µ+λz +ǫ. where ǫ and z are independent. Let s work out exactly what distribution our model defines. Our random variables z and x have a joint Gaussian distribution [ ] z N(µ x zx,σ). We will now find µ zx and Σ. We know that E[z] = 0, from the fact that z N(0,I). Also, we have that Putting these together, we obtain E[x] = E[µ+Λz +ǫ] = µ+λe[z]+e[ǫ] = µ. µ zx = [ 0 µ Next, to find, Σ, we need to calculate Σ zz = E[(z E[z])(z E[z]) T ] (the upper-left block of Σ), Σ zx = E[(z E[z])(x E[x]) T ] (upper-right block), and Σ xx = E[(x E[x])(x E[x]) T ] (lower-right block). Now, since z N(0,I), we easily find that Σ zz = Cov(z) = I. Also, E[(z E[z])(x E[x]) T ] = E[z(µ+Λz +ǫ µ) T ] ] = E[zz T ]Λ T +E[zǫ T ] = Λ T. In the last step, we used the fact that E[zz T ] = Cov(z) (since z has zero mean), and E[zǫ T ] = E[z]E[ǫ T ] = 0 (since z and ǫ are independent, and

6 6 hence the expectation of their product is the product of their expectations). Similarly, we can find Σ xx as follows: E[(x E[x])(x E[x]) T ] = E[(µ+Λz +ǫ µ)(µ+λz +ǫ µ) T ] = E[Λzz T Λ T +ǫz T Λ T +Λzǫ T +ǫǫ T ] = ΛE[zz T ]Λ T +E[ǫǫ T ] = ΛΛ T +Ψ. Putting everything together, we therefore have that [ ] ([ ] [ z 0 I Λ T N, x µ Λ ΛΛ T +Ψ ]). (3) Hence, we also see that the marginal distribution of x is given by x N(µ,ΛΛ T +Ψ). Thus, given a training set {x (i) ;i = 1,...,m}, we can write down the log likelihood of the parameters: l(µ,λ,ψ) = log m ( 1 exp 1 ) (2π) n/2 ΛΛ T +Ψ 1/2 2 (x(i) µ) T (ΛΛ T +Ψ) 1 (x (i) µ). To perform maximum likelihood estimation, we would like to maximize this quantity with respect to the parameters. But maximizing this formula explicitly is hard (try it yourself), and we are aware of no algorithm that does so in closed-form. So, we will instead use to the EM algorithm. In the next section, we derive EM for factor analysis. 4 EM for factor analysis The derivation for the E-step is easy. We need to compute Q i (z (i) ) = p(z (i) x (i) ;µ,λ,ψ). By substituting the distribution given in Equation (3) into the formulas (1-2) used for finding the conditional distribution of a Gaussian, we find that z (i) x (i) ;µ,λ,ψ N(µ z (i) x (i),σ z (i) x (i)), where µ z (i) x (i) = ΛT (ΛΛ T +Ψ) 1 (x (i) µ), Σ z (i) x (i) = I ΛT (ΛΛ T +Ψ) 1 Λ. So, using these definitions for µ z (i) x and Σ (i) z (i) x (i), we have Q i (z (i) 1 ) = (2π) k/2 Σ z (i) x ( exp (i) 1/2 1 2 (z(i) µ z (i) x (i))t Σ 1 z (i) x (i) (z (i) µ z (i) x (i)) ).

7 7 Let s now work out the M-step. Here, we need to maximize Q i (z (i) )log p(x(i),z (i) ;µ,λ,ψ) dz (i) (4) z Q (i) i (z (i) ) with respect to the parameters µ,λ,ψ. We will work out only the optimization with respect to Λ, and leave the derivations of the updates for µ and Ψ as an exercise to the reader. We can simplify Equation (4) as follows: Q i (z (i) ) [ logp(x (i) z (i) ;µ,λ,ψ)+logp(z (i) ) logq i (z (i) ) ] dz (i) (5) z (i) [ = E z (i) Q i logp(x (i) z (i) ;µ,λ,ψ)+logp(z (i) ) logq i (z (i) ) ] (6) Here, the z (i) Q i subscript indicates that the expectation is with respect to z (i) drawn from Q i. In the subsequent development, we will omit this subscript when there is no risk of ambiguity. Dropping terms that do not depend on the parameters, we find that we need to maximize: E [ logp(x (i) z (i) ;µ,λ,ψ) ] = = [ E log ( 1 exp 1 )] (2π) n/2 Ψ 1/2 2 (x(i) µ Λz (i) ) T Ψ 1 (x (i) µ Λz (i) ) ] [ E 1 2 log Ψ n 2 log(2π) 1 2 (x(i) µ Λz (i) ) T Ψ 1 (x (i) µ Λz (i) ) Let s maximize this with respect to Λ. Only the last term above depends on Λ. Taking derivatives, and using the facts that tra = a (for a R), trab = trba, and A traba T C = CAB +C T AB, we get: m [ ] 1 Λ E 2 (x(i) µ Λz (i) ) T Ψ 1 (x (i) µ Λz (i) ) [ = Λ E tr 1 ] 2 z(i)t Λ T Ψ 1 Λz (i) +trz (i)t Λ T Ψ 1 (x (i) µ) = = Λ E [ tr 12 ] ΛT Ψ 1 Λz (i) z (i)t +trλ T Ψ 1 (x (i) µ)z (i)t E [ Ψ 1 Λz (i) z (i)t +Ψ 1 (x (i) µ)z (i)t]

8 8 Setting this to zero and simplifying, we get: ΛE z (i) Q i [z (i) z (i)t] = (x (i) µ)e z (i) Q i [z (i)t]. Hence, solving for Λ, we obtain ( m [ m Λ = (x (i) µ)e z (i) Q i z (i)t])( E z (i) Q i [ z (i) z (i)t]) 1. (7) It is interesting to note the close relationship between this equation and the normal equation that we d derived for least squares regression, θ T = (y T X)(X T X) 1. The analogy is that here, the x s are a linear function of the z s (plus noise). Given the guesses for z that the E-step has found, we will now try to estimate the unknown linearity Λ relating the x s and z s. It is therefore no surprise that we obtain something similar to the normal equation. There is, however, one important difference between this and an algorithm that performs least squares using just the best guesses of the z s; we will see this difference shortly. To complete our M-step update, let s work out the values of the expectations in Equation (7). From our definition of Q i being Gaussian with mean µ z (i) x and covariance Σ (i) z (i) x (i), we easily find E z (i) Q i [z (i)t] = µ T z (i) x (i) E z (i) Q i [z (i) z (i)t] = µ z (i) x (i)µt z (i) x (i) +Σ z (i) x (i). The latter comes from the fact that, for a random variable Y, Cov(Y) = E[YY T ] E[Y]E[Y] T, and hence E[YY T ] = E[Y]E[Y] T +Cov(Y). Substituting this back into Equation (7), we get the M-step update for Λ: ( m )( m ) 1 Λ = (x (i) µ)µ T z (i) x µ (i) z (i) x (i)µt z (i) x +Σ (i) z (i) x. (8) (i) It is important to note the presence of the Σ z (i) x (i) on the right hand side of this equation. This is the covariance in the posterior distribution p(z (i) x (i) ) of z (i) give x (i), and the M-step must take into account this uncertainty

9 9 about z (i) in the posterior. A common mistake in deriving EM is to assume that in the E-step, we need to calculate only expectation E[z] of the latent random variable z, and then plug that into the optimization in the M-step everywhere z occurs. While this worked for simple problems such as the mixture of Gaussians, in our derivation for factor analysis, we needed E[zz T ] as well E[z]; and as we saw, E[zz T ] and E[z]E[z] T differ by the quantity Σ z x. Thus, the M-step update must take into account the covariance of z in the posterior distribution p(z (i) x (i) ). Lastly, we can also find the M-step optimizations for the parameters µ and Ψ. It is not hard to show that the first is given by µ = 1 m x (i). Since this doesn t change as the parameters are varied(i.e., unlike the update for Λ, the right hand side does not depend on Q i (z (i) ) = p(z (i) x (i) ;µ,λ,ψ), which in turn depends on the parameters), this can be calculated just once and needs not be further updated as the algorithm is run. Similarly, the diagonal Ψ can be found by calculating Φ = 1 m x (i) x (i)t x (i) µ T z (i) x Λ T Λµ (i) z (i) x (i)x(i)t +Λ(µ z (i) x (i)µt z (i) x +Σ (i) z (i) x (i))λt, and setting Ψ ii = Φ ii (i.e., letting Ψ be the diagonal matrix containing only the diagonal entries of Φ).

### Sections 2.11 and 5.8

Sections 211 and 58 Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1/25 Gesell data Let X be the age in in months a child speaks his/her first word and

### Lecture 9: Continuous

CSC2515 Fall 2007 Introduction to Machine Learning Lecture 9: Continuous Latent Variable Models 1 Example: continuous underlying variables What are the intrinsic latent dimensions in these two datasets?

### Hypothesis Testing in the Classical Regression Model

LECTURE 5 Hypothesis Testing in the Classical Regression Model The Normal Distribution and the Sampling Distributions It is often appropriate to assume that the elements of the disturbance vector ε within

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### Lecture 8: Signal Detection and Noise Assumption

ECE 83 Fall Statistical Signal Processing instructor: R. Nowak, scribe: Feng Ju Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(, σ I n n and S = [s, s,...,

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### The Bivariate Normal Distribution

The Bivariate Normal Distribution This is Section 4.7 of the st edition (2002) of the book Introduction to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Principal components analysis

CS229 Lecture notes Andrew Ng Part XI Principal components analysis In our discussion of factor analysis, we gave a way to model data x R n as approximately lying in some k-dimension subspace, where k

### Lecture Notes 3 Random Vectors. Specifying a Random Vector. Mean and Covariance Matrix. Coloring and Whitening. Gaussian Random Vectors

Lecture Notes 3 Random Vectors Specifying a Random Vector Mean and Covariance Matrix Coloring and Whitening Gaussian Random Vectors EE 278B: Random Vectors 3 Specifying a Random Vector Let X,X 2,...,X

### Introduction to General and Generalized Linear Models

Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby

### Math 313 Lecture #10 2.2: The Inverse of a Matrix

Math 1 Lecture #10 2.2: The Inverse of a Matrix Matrix algebra provides tools for creating many useful formulas just like real number algebra does. For example, a real number a is invertible if there is

### ( % . This matrix consists of \$ 4 5 " 5' the coefficients of the variables as they appear in the original system. The augmented 3 " 2 2 # 2 " 3 4&

Matrices define matrix We will use matrices to help us solve systems of equations. A matrix is a rectangular array of numbers enclosed in parentheses or brackets. In linear algebra, matrices are important

### Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0

Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0 This primer provides an overview of basic concepts and definitions in probability and statistics. We shall

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Matrices provide a compact notation for expressing systems of equations or variables. For instance, a linear function might be written as: b k

MATRIX ALGEBRA FOR STATISTICS: PART 1 Matrices provide a compact notation for expressing systems of equations or variables. For instance, a linear function might be written as: y = x 1 b 1 + x 2 + x 3

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Factor analysis. Angela Montanari

Factor analysis Angela Montanari 1 Introduction Factor analysis is a statistical model that allows to explain the correlations between a large number of observed correlated variables through a small number

### ELEC-E8104 Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems

Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems Minimum Mean Square Error (MMSE) MMSE estimation of Gaussian random vectors Linear MMSE estimator for arbitrarily distributed

### Joint Probability Distributions and Random Samples (Devore Chapter Five)

Joint Probability Distributions and Random Samples (Devore Chapter Five) 1016-345-01 Probability and Statistics for Engineers Winter 2010-2011 Contents 1 Joint Probability Distributions 1 1.1 Two Discrete

### OLS in Matrix Form. Let y be an n 1 vector of observations on the dependent variable.

OLS in Matrix Form 1 The True Model Let X be an n k matrix where we have observations on k independent variables for n observations Since our model will usually contain a constant term, one of the columns

### Example: Credit card default, we may be more interested in predicting the probabilty of a default than classifying individuals as default or not.

Statistical Learning: Chapter 4 Classification 4.1 Introduction Supervised learning with a categorical (Qualitative) response Notation: - Feature vector X, - qualitative response Y, taking values in C

### What is the interpretation of R 2?

What is the interpretation of R 2? Karl G. Jöreskog October 2, 1999 Consider a regression equation between a dependent variable y and a set of explanatory variables x'=(x 1, x 2,..., x q ): or in matrix

### Random Vectors and the Variance Covariance Matrix

Random Vectors and the Variance Covariance Matrix Definition 1. A random vector X is a vector (X 1, X 2,..., X p ) of jointly distributed random variables. As is customary in linear algebra, we will write

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### The Exponential Family

The Exponential Family David M. Blei Columbia University November 3, 2015 Definition A probability density in the exponential family has this form where p.x j / D h.x/ expf > t.x/ a./g; (1) is the natural

### 7 Time series analysis

7 Time series analysis In Chapters 16, 17, 33 36 in Zuur, Ieno and Smith (2007), various time series techniques are discussed. Applying these methods in Brodgar is straightforward, and most choices are

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### 1 Short Introduction to Time Series

ECONOMICS 7344, Spring 202 Bent E. Sørensen January 24, 202 Short Introduction to Time Series A time series is a collection of stochastic variables x,.., x t,.., x T indexed by an integer value t. The

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Definition A square matrix M is invertible (or nonsingular) if there exists a matrix M 1 such that

0. Inverse Matrix Definition A square matrix M is invertible (or nonsingular) if there exists a matrix M such that M M = I = M M. Inverse of a 2 2 Matrix Let M and N be the matrices: a b d b M =, N = c

### Introduction to latent variable models

Introduction to latent variable models lecture 1 Francesco Bartolucci Department of Economics, Finance and Statistics University of Perugia, IT bart@stat.unipg.it Outline [2/24] Latent variables and their

### LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

### Auxiliary Variables in Mixture Modeling: 3-Step Approaches Using Mplus

Auxiliary Variables in Mixture Modeling: 3-Step Approaches Using Mplus Tihomir Asparouhov and Bengt Muthén Mplus Web Notes: No. 15 Version 8, August 5, 2014 1 Abstract This paper discusses alternatives

### Clustering - example. Given some data x i X Find a partitioning of the data into k disjunctive clusters Example: k-means clustering

Clustering - example Graph Mining and Graph Kernels Given some data x i X Find a partitioning of the data into k disjunctive clusters Example: k-means clustering x!!!!8!!! 8 x 1 1 Clustering - example

### Some probability and statistics

Appendix A Some probability and statistics A Probabilities, random variables and their distribution We summarize a few of the basic concepts of random variables, usually denoted by capital letters, X,Y,

### Solving Challenging Non-linear Regression Problems by Manipulating a Gaussian Distribution

Solving Challenging Non-linear Regression Problems by Manipulating a Gaussian Distribution Sheffield Gaussian Process Summer School, 214 Carl Edward Rasmussen Department of Engineering, University of Cambridge

### Gaussian Processes in Machine Learning

Gaussian Processes in Machine Learning Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics, 72076 Tübingen, Germany carl@tuebingen.mpg.de WWW home page: http://www.tuebingen.mpg.de/ carl

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Classification Problems

Classification Read Chapter 4 in the text by Bishop, except omit Sections 4.1.6, 4.1.7, 4.2.4, 4.3.3, 4.3.5, 4.3.6, 4.4, and 4.5. Also, review sections 1.5.1, 1.5.2, 1.5.3, and 1.5.4. Classification Problems

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### Course: Model, Learning, and Inference: Lecture 5

Course: Model, Learning, and Inference: Lecture 5 Alan Yuille Department of Statistics, UCLA Los Angeles, CA 90095 yuille@stat.ucla.edu Abstract Probability distributions on structured representation.

### Gaussian Conjugate Prior Cheat Sheet

Gaussian Conjugate Prior Cheat Sheet Tom SF Haines 1 Purpose This document contains notes on how to handle the multivariate Gaussian 1 in a Bayesian setting. It focuses on the conjugate prior, its Bayesian

### 1 Eigenvalues and Eigenvectors

Math 20 Chapter 5 Eigenvalues and Eigenvectors Eigenvalues and Eigenvectors. Definition: A scalar λ is called an eigenvalue of the n n matrix A is there is a nontrivial solution x of Ax = λx. Such an x

### Statistical Machine Learning from Data

Samy Bengio Statistical Machine Learning from Data 1 Statistical Machine Learning from Data Gaussian Mixture Models Samy Bengio IDIAP Research Institute, Martigny, Switzerland, and Ecole Polytechnique

### Lecture 10: Invertible matrices. Finding the inverse of a matrix

Lecture 10: Invertible matrices. Finding the inverse of a matrix Danny W. Crytser April 11, 2014 Today s lecture Today we will Today s lecture Today we will 1 Single out a class of especially nice matrices

### Quadratic forms Cochran s theorem, degrees of freedom, and all that

Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us

### P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

### Principal Component Analysis Application to images

Principal Component Analysis Application to images Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering, Department of Cybernetics Center for Machine Perception http://cmp.felk.cvut.cz/

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### Notes on Orthogonal and Symmetric Matrices MENU, Winter 2013

Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,

### Random Variables, Expectation, Distributions

Random Variables, Expectation, Distributions CS 5960/6960: Nonparametric Methods Tom Fletcher January 21, 2009 Review Random Variables Definition A random variable is a function defined on a probability

### Mixtures of Robust Probabilistic Principal Component Analyzers

Mixtures of Robust Probabilistic Principal Component Analyzers Cédric Archambeau, Nicolas Delannay 2 and Michel Verleysen 2 - University College London, Dept. of Computer Science Gower Street, London WCE

### 2.1: MATRIX OPERATIONS

.: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

### 3. The Multivariate Normal Distribution

3. The Multivariate Normal Distribution 3.1 Introduction A generalization of the familiar bell shaped normal density to several dimensions plays a fundamental role in multivariate analysis While real data

### CS 688 Pattern Recognition Lecture 4. Linear Models for Classification

CS 688 Pattern Recognition Lecture 4 Linear Models for Classification Probabilistic generative models Probabilistic discriminative models 1 Generative Approach ( x ) p C k p( C k ) Ck p ( ) ( x Ck ) p(

### November 28 th, Carlos Guestrin. Lower dimensional projections

PCA Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University November 28 th, 2007 Lower dimensional projections Rather than picking a subset of the features, we can new features that are combinations

### University of Ljubljana Doctoral Programme in Statistics Methodology of Statistical Research Written examination February 14 th, 2014.

University of Ljubljana Doctoral Programme in Statistics ethodology of Statistical Research Written examination February 14 th, 2014 Name and surname: ID number: Instructions Read carefully the wording

### SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation

SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu September 19, 2015 Outline

### Recitation 4. 24xy for 0 < x < 1, 0 < y < 1, x + y < 1 0 elsewhere

Recitation. Exercise 3.5: If the joint probability density of X and Y is given by xy for < x

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### Practical Numerical Training UKNum

Practical Numerical Training UKNum 7: Systems of linear equations C. Mordasini Max Planck Institute for Astronomy, Heidelberg Program: 1) Introduction 2) Gauss Elimination 3) Gauss with Pivoting 4) Determinants

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### 4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

### A SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS

A SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS Eusebio GÓMEZ, Miguel A. GÓMEZ-VILLEGAS and J. Miguel MARÍN Abstract In this paper it is taken up a revision and characterization of the class of

### Wooldridge, Introductory Econometrics, 4th ed. Chapter 15: Instrumental variables and two stage least squares

Wooldridge, Introductory Econometrics, 4th ed. Chapter 15: Instrumental variables and two stage least squares Many economic models involve endogeneity: that is, a theoretical relationship does not fit

### The basic unit in matrix algebra is a matrix, generally expressed as: a 11 a 12. a 13 A = a 21 a 22 a 23

(copyright by Scott M Lynch, February 2003) Brief Matrix Algebra Review (Soc 504) Matrix algebra is a form of mathematics that allows compact notation for, and mathematical manipulation of, high-dimensional

### , then the form of the model is given by: which comprises a deterministic component involving the three regression coefficients (

Multiple regression Introduction Multiple regression is a logical extension of the principles of simple linear regression to situations in which there are several predictor variables. For instance if we

### Matrix Algebra 2.3 CHARACTERIZATIONS OF INVERTIBLE MATRICES Pearson Education, Inc.

2 Matrix Algebra 2.3 CHARACTERIZATIONS OF INVERTIBLE MATRICES Theorem 8: Let A be a square matrix. Then the following statements are equivalent. That is, for a given A, the statements are either all true

### Correlation in Random Variables

Correlation in Random Variables Lecture 11 Spring 2002 Correlation in Random Variables Suppose that an experiment produces two random variables, X and Y. What can we say about the relationship between

### EC 6310: Advanced Econometric Theory

EC 6310: Advanced Econometric Theory July 2008 Slides for Lecture on Bayesian Computation in the Nonlinear Regression Model Gary Koop, University of Strathclyde 1 Summary Readings: Chapter 5 of textbook.

### The Simple Linear Regression Model: Specification and Estimation

Chapter 3 The Simple Linear Regression Model: Specification and Estimation 3.1 An Economic Model Suppose that we are interested in studying the relationship between household income and expenditure on

### The Optimality of Naive Bayes

The Optimality of Naive Bayes Harry Zhang Faculty of Computer Science University of New Brunswick Fredericton, New Brunswick, Canada email: hzhang@unbca E3B 5A3 Abstract Naive Bayes is one of the most

### Least Squares Estimation

Least Squares Estimation SARA A VAN DE GEER Volume 2, pp 1041 1045 in Encyclopedia of Statistics in Behavioral Science ISBN-13: 978-0-470-86080-9 ISBN-10: 0-470-86080-4 Editors Brian S Everitt & David

### Orthogonal Projections

Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Component Ordering in Independent Component Analysis Based on Data Power

Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals

### Lecture Quantitative Finance

Lecture Quantitative Finance Spring 2011 Prof. Dr. Erich Walter Farkas Lecture 12: May 19, 2011 Chapter 8: Estimating volatility and correlations Prof. Dr. Erich Walter Farkas Quantitative Finance 11:

### Adding vectors We can do arithmetic with vectors. We ll start with vector addition and related operations. Suppose you have two vectors

1 Chapter 13. VECTORS IN THREE DIMENSIONAL SPACE Let s begin with some names and notation for things: R is the set (collection) of real numbers. We write x R to mean that x is a real number. A real number

### i=1 In practice, the natural logarithm of the likelihood function, called the log-likelihood function and denoted by

Statistics 580 Maximum Likelihood Estimation Introduction Let y (y 1, y 2,..., y n be a vector of iid, random variables from one of a family of distributions on R n and indexed by a p-dimensional parameter

### State Space Time Series Analysis

State Space Time Series Analysis p. 1 State Space Time Series Analysis Siem Jan Koopman http://staff.feweb.vu.nl/koopman Department of Econometrics VU University Amsterdam Tinbergen Institute 2011 State

### Lecture Notes: Matrix Inverse. 1 Inverse Definition

Lecture Notes: Matrix Inverse Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Inverse Definition We use I to represent identity matrices,

### Linear Systems. Singular and Nonsingular Matrices. Find x 1, x 2, x 3 such that the following three equations hold:

Linear Systems Example: Find x, x, x such that the following three equations hold: x + x + x = 4x + x + x = x + x + x = 6 We can write this using matrix-vector notation as 4 {{ A x x x {{ x = 6 {{ b General

### SYSTEMS OF REGRESSION EQUATIONS

SYSTEMS OF REGRESSION EQUATIONS 1. MULTIPLE EQUATIONS y nt = x nt n + u nt, n = 1,...,N, t = 1,...,T, x nt is 1 k, and n is k 1. This is a version of the standard regression model where the observations

### LECTURE 1 I. Inverse matrices We return now to the problem of solving linear equations. Recall that we are trying to find x such that IA = A

LECTURE I. Inverse matrices We return now to the problem of solving linear equations. Recall that we are trying to find such that A = y. Recall: there is a matri I such that for all R n. It follows that

### A Introduction to Matrix Algebra and Principal Components Analysis

A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2 Today s Class An introduction to matrix algebra

### Direct Methods for Solving Linear Systems. Linear Systems of Equations

Direct Methods for Solving Linear Systems Linear Systems of Equations Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University

### Recall that two vectors in are perpendicular or orthogonal provided that their dot

Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal

### 1.5. Factorisation. Introduction. Prerequisites. Learning Outcomes. Learning Style

Factorisation 1.5 Introduction In Block 4 we showed the way in which brackets were removed from algebraic expressions. Factorisation, which can be considered as the reverse of this process, is dealt with

### problem arises when only a non-random sample is available differs from censored regression model in that x i is also unobserved

4 Data Issues 4.1 Truncated Regression population model y i = x i β + ε i, ε i N(0, σ 2 ) given a random sample, {y i, x i } N i=1, then OLS is consistent and efficient problem arises when only a non-random

### What is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference

0. 1. Introduction and probability review 1.1. What is Statistics? What is Statistics? Lecture 1. Introduction and probability review There are many definitions: I will use A set of principle and procedures

### Multivariate Analysis (Slides 13)

Multivariate Analysis (Slides 13) The final topic we consider is Factor Analysis. A Factor Analysis is a mathematical approach for attempting to explain the correlation between a large set of variables