Lecture Notes 3 Random Vectors. Specifying a Random Vector. Mean and Covariance Matrix. Coloring and Whitening. Gaussian Random Vectors
|
|
- Eustace Gordon
- 7 years ago
- Views:
Transcription
1 Lecture Notes 3 Random Vectors Specifying a Random Vector Mean and Covariance Matrix Coloring and Whitening Gaussian Random Vectors EE 278B: Random Vectors 3 Specifying a Random Vector Let X,X 2,...,X n be random variables defined on the same probability space. We define a random vector (RV) as X X = X 2. X is completely specified by its joint cdf for x =(x,x 2,...,x n ): X n F X (x) =P{X x,x 2 x 2,...,X n x n }, x R n If X is continuous, i.e., F X (x) is a continuous function of x, thenx can be specified by its joint pdf: f X (x) =f X,X 2,...,X n (x,x 2,...,x n ), If X is discrete then it can be specified by its joint pmf: p X (x) =p X,X 2,...,X n (x,x 2,...,x n ), x R n x X n EE 278B: Random Vectors 3 2
2 Amarginalcdf(pdf,pmf)isthejointcdf(pdf,pmf)forasubset of {X,...,X n };e.g.,for X = X the marginals are X 2 X 3 f X (x ),f X2 (x 2 ),f X3 (x 3 ) f X,X 2 (x,x 2 ),f X,X 3 (x,x 3 ),f X2,X 3 (x 2,x 3 ) The marginals can be obtained from the joint in the usual way. For the previous example, F X (x )= lim F X(x,x 2,x 3 ) x 2,x 3 f X,X 2 (x,x 2 )= f X,X 2,X 3 (x,x 2,x 3 ) dx 3 EE 278B: Random Vectors 3 3 Conditional cdf (pdf, pmf) can also be defined in the usual way. E.g.,the conditional pdf of X n k+ =(X k+,...,x n ) given X k =(X,...,X k ) is Chain Rule: Wecanwrite f X n k+ X k(xn k+ x k )= f X(x,x 2,...,x n ) f X k(x,x 2,...,x k ) = f X(x) f X k(x k ) f X (x) =f X (x )f X2 X (x 2 x )f X3 X,X 2 (x 3 x,x 2 ) f Xn X n (x n x n ) Proof: By induction. The chain rule holds for n =2by definition of conditional pdf. Now suppose it is true for n. Then f X (x) =f X n (x n )f Xn X n (x n x n ) = f X (x )f X2 X (x 2 x ) f Xn X n 2(x n x n 2 )f Xn X n (x n x n ), which completes the proof EE 278B: Random Vectors 3 4
3 Independence and Conditional Independence Independence is defined in the usual way; e.g., X,X 2,...,X n are independent if n f X (x) = f Xi (x i ) for all (x,...,x n ) i= Important special case, i.i.d. r.v.s: X,X 2,...,X n are said to be independent, identically distributed (i.i.d.) if they are independent and have the same marginals Example: if we flip a coin n times independently, we generate i.i.d. Bern(p) r.v.s. X,X 2,...,X n R.v.s X and X 3 are said to be conditionally independent given X 2 if f X,X 3 X 2 (x,x 3 x 2 )=f X X 2 (x x 2 )f X3 X 2 (x 3 x 2 ) for all (x,x 2,x 3 ) Conditional independence neither implies nor is implied by independence; X and X 3 independent given X 2 does not mean that X and X 3 are independent (or vice versa) EE 278B: Random Vectors 3 5 Example: Coin with random bias. GivenacoinwithrandombiasP f P (p), flip it n times independently to generate the r.v.s X,X 2,...,X n,where X i =if i-th flip is heads, 0 otherwise X,X 2,...,X n are not independent However, X,X 2,...,X n are conditionally independent given P ;infact,they are i.i.d. Bern(p) for every P = p Example: Additive noise channel. Consideranadditivenoisechannelwithsignal X, noisez, andobservationy = X + Z, wherex and Z are independent Although X and Z are independent, they are not in general conditionally independent give Y EE 278B: Random Vectors 3 6
4 Mean and Covariance Matrix The mean of the random vector X is defined as E(X) = [ E(X ) E(X 2 ) E(X n ) ] T Denote the covariance between X i and X j, Cov(X i,x j ),byσ ij (so the variance of X i is denoted by σ ii, Var(X i ),orσ 2 X i ) The covariance matrix of X is defined as σ σ 2 σ n Σ X = σ 2 σ 22 σ 2n σ n σ n2 σ nn For n =2,wecanusethedefinitionofcorrelationcoefficienttoobtain [ σ σ Σ X = 2 σ 2 ] X = ρ X,X 2 σ X σ X2 σ 2 σ 22 ρ X,X 2 σ X σ X2 σx 2 2 EE 278B: Random Vectors 3 7 Properties of Covariance Matrix Σ X Σ X is real and symmetric (since σ ij = σ ji ) Σ X is positive semidefinite, i.e.,thequadratic form a T Σ X a 0 for every real vector a Equivalently, all the eigenvalues of Σ X are nonnegative, and also all leading principal minors are nonnegative To show that Σ X is positive semidefinite we write Σ X =E [ (X E(X))(X E(X)) T ], i.e., as the expectation of an outer product. Thus a T Σ X a = a T E [ (X E(X))(X E(X)) T ] a =E [ a T (X E(X))(X E(X)) T a ] =E [ (a T (X E(X))) 2] 0 EE 278B: Random Vectors 3 8
5 Which of the Following Can Be a Covariance Matrix? EE 278B: Random Vectors 3 9 Coloring and Whitening Square root of covariance matrix: LetΣ be a covariance matrix. Then there exists an n n matrix Σ /2 such that Σ=Σ /2 (Σ /2 ) T.ThematrixΣ /2 is called the square root of Σ Coloring: LetX be white RV, i.e., has zero mean and Σ X = I. Assumewithout loss of generality that a = Let Σ be a covariance matrix, then the RV Y =Σ /2 X has covariance matrix Σ (why?) Hence we can generate a RV with any prescribed covariance from awhiterv Whitening: GivenazeromeanRVY with nonsingular covariance matrix Σ, then the RV X =Σ /2 Y is white Hence, we can generate a white RV from any RV with nonsingular covariance matrix Coloring and whitening have applications in simulations, detection, and estimation EE 278B: Random Vectors 3 0
6 Finding Square Root of Σ For convenience, we assume throughout that Σ is nonsingular Since Σ is symmetric, it has n real eigenvalues λ,λ 2,...,λ n and n corresponding orthogonal eigenvectors u, u 2,...,u n Further, since Σ is positive definite, the eigenvalues are all positive Thus, we have Σu i = λ i u i, u T i u j =0 λ i > 0, i=, 2,...,n for every i j Without loss of generality assume that the u i vectors are unit vectors The first set of equations can be rewritten in the matrix form where ΣU = UΛ, U =[u u 2... u n ] and Λ is a diagonal matrix with diagonal elements λ i EE 278B: Random Vectors 3 Note that U is a unitary matrix (U T U = UU T = I), hence Σ=UΛU T and the square root of Σ is Σ /2 = UΛ /2, where Λ /2 is a diagonal matrix with diagonal elements λ /2 i The inverse of the square root is straightforward to find as Σ /2 =Λ /2 U T Example: Let 2 Σ= 3 To find the eigenvalues of Σ, wefindtherootsofthepolynomialequation which gives λ =3.62, λ 2 =.38 To find the eigenvectors, consider [ 2 3 det(σ λi) =λ 2 5λ +5=0, ][ u u 2 ] u =3.62, u 2 EE 278B: Random Vectors 3 2
7 and u 2 + u 2 2 =,whichyields u = Similarly, we can find the second eigenvector 0.85 u 2 = 0.53 Hence, Σ /2 = [ ] = The inverse of the square root is [ ] Σ /2 / = 0 / = [ ] [ 0.28 ] Geometric interpretation: To generate a RV Y with covariance matrix Σ from white RV X, weusethetransformationy = UΛ /2 X Equivalently, we first scale each component of X to obtain the RV Z =Λ /2 X We then rotate Z using U to obtain Y = UZ EE 278B: Random Vectors 3 3 Cholesky Decomposition Σ has many square roots: If Σ /2 is a square root, then for any unitary matrix V, Σ /2 V is also a square root since Σ /2 VV T (Σ /2 ) T =Σ The Cholesky decomposition is an efficient algorithm for computing lower triangle square root that can be used to perform coloring causally (sequentially) For n =3,wewanttofindalowertrianglematrix(squareroot)A such that Σ= σ σ 2 σ 3 σ 2 σ 22 σ 23 = a 0 0 a 2 a 22 0 a a 2 a 3 0 a 22 a 32 σ 3 σ 32 σ 33 a 3 a 32 a a 33 The elements of A are computed in a raster scan manner: a : σ = a 2 a = σ a 2 : σ 2 = a 2 a a 2 = σ 2 /a a 22 : σ 22 = a a 2 22 a 22 = σ 22 a 2 2 a 3 : σ 3 = a a 3 a 3 = σ 3 /a EE 278B: Random Vectors 3 4
8 a 32 : σ 32 = a 2 a 3 + a 22 a 32 a 32 = σ 32 a 2 a 3 )/a 22 a 33 : σ 33 = a a a 2 33 a 33 = σ 33 a 2 3 a2 32 The inverse of a lower triangle square root is also lower triangular Coloring and whitening summary: Coloring: X Σ /2 Y Whitening: Σ X = I Σ Y =Σ Y Σ /2 X Σ Y =Σ Σ X = I Lower triangle square root and its inverse can be efficiently computed using Cholesky decomposition EE 278B: Random Vectors 3 5 Gaussian Random Vectors ArandomvectorX =(X,...,X n ) is a Gaussian random vector (GRV) (or X,X 2,...,X n are jointly Gaussian r.v.s) if the joint pdf is of the form f X (x) = e (2π) n 2 Σ 2 2 (x µ)t Σ (x µ), where µ is the mean and Σ is the covariance matrix of X, and Σ > 0, i.e.,σ is positive definite Verify that this joint pdf is the same as the case n =2from Lecture Notes 2 Notation: X N(µ, Σ) denotes a GRV with given mean and covariance matrix Since Σ is positive definite, Σ is positive definite. Thus if x µ 0, (x µ) T Σ (x µ) > 0, which means that the contours of equal pdf are ellipsoids The GRV X N(0,aI), wherei is the identity matrix and a>0, iscalled white; itscontoursofequaljointpdfarespherescenteredattheorigin EE 278B: Random Vectors 3 6
9 Properties of GRVs Property : ForaGRV,uncorrelationimpliesindependence This can be verified by substituting σ ij =0for all i j in the joint pdf. Then Σ becomes diagonal and so does Σ,andthejointpdfreducestothe product of the marginals X i N(µ i,σ ii ) For the white GRV X N(0,aI), ther.v.sarei.i.d.n (0,a) Property 2: LineartransformationofaGRVyieldsaGRV,i.e.,givenany m n matrix A, wherem n and A has full rank m, then Y = AX N(Aµ, AΣA T ) Example: Let Find the joint pdf of X N Y = ( 0, ) 2 3 X 0 EE 278B: Random Vectors 3 7 Solution: From Property 2, we conclude that ( ) 2 Y N 0, = N Before we prove Property 2, let us show that ( 0, E(Y) =Aµ and Σ Y = AΣA T ) These results follow from linearity of expectation. First, expectation: E(Y) =E(AX) =A E(X) =Aµ Next consider the covariance matrix: Σ Y =E [ (Y E(Y))(Y E(Y)) T ] =E [ (AX Aµ)(AX Aµ) T ] = A E [ (X µ)(x µ) T ] A T = AΣA T Of course this is not sufficient to show that Y is a GRV we must also show that the joint pdf has the right form We do so using the characteristic function for a random vector EE 278B: Random Vectors 3 8
10 Definition: IfX f X (x), thecharacteristicfunctionofx is ( ) Φ X (ω) =E e iωt X, where ω is an n-dimensional real valued vector and i = Thus Φ X (ω) =... f X (x)e iωt x dx This is the inverse of the multi-dimensional Fourier transform of f X (x), which implies that there is a one-to-one correspondence between Φ X (ω) and f X (x). The joint pdf can be found by taking the Fourier transform of Φ X (ω), i.e., f X (x) =... (2π) nφ X(ω)e iω T x dω Example: The characteristic function for X N(µ, σ 2 ) is and for a GRV X N(µ, Σ), Φ X (ω) =e 2 ω2 σ 2 + iµω, Φ X (ω) =e 2 ωt Σω + iω T µ EE 278B: Random Vectors 3 9 Now let s go back to proving Property 2 Since A is an m n matrix, Y = AX and ω are m-dimensional. Therefore the characteristic function of Y is ( ) Φ Y (ω) =E e iωt Y ( ) =E e iωt AX Thus Y = AX N(Aµ,AΣA T ) =Φ X (A T ω) = e 2 (AT ω) T Σ(A T ω)+iω T Aµ = e 2 ωt (AΣA T )ω + iω T Aµ An equivalent definition of GRV: X is a GRV iff for any real vector a 0,the r.v. Y = a T X is Gaussian (see HW for proof) Whitening transforms a GRV to a white GRV; conversely, coloring transforms a white GRV to a GRV with prescribed covariance matrix EE 278B: Random Vectors 3 20
11 Property 3: MarginalsofaGRVareGaussian,i.e.,ifX is GRV then for any subset {i,i 2,...,i k } {, 2,...,n} of indexes, the RV is a GRV Y = X i X i2. X ik To show this we use Property 2. For example, let n =3and Y = We can express Y as a linear transformation of X: 0 0 Y = X X X = X 3 Therefore Y N ([ µ µ 3 X 3 ] ) σ σ, 3 σ 3 σ 33 [ X As we have seen in Lecture Notes 2, the converse of Property 3 does not hold in general, i.e., Gaussian marginals do not necessarily mean that the r.v.s are jointly Gaussian X 3 ] EE 278B: Random Vectors 3 2 Property 4: ConditionalsofaGRVareGaussian,morespecifically,if X = X N µ, Σ Σ 2, Σ 2 Σ 22 X 2 where X is a k-dim RV and X 2 is an n k-dim RV, then X 2 {X = x} N ( Σ 2 Σ (x µ )+µ 2, Σ 22 Σ 2 Σ Σ ) 2 Compare this to the case of n =2and k =: ( ) σ2 X 2 {X = x} N (x µ )+µ 2,σ 22 σ2 2 σ σ µ 2 Example: X 2 X 2 N 2, X 3 EE 278B: Random Vectors 3 22
12 From Property 4, it follows that E(X 2 X = x) = Σ {X2 X =x} = = [ [ 2 2 2x (x ) + = ] 2] x [2 ] The proof of Property 4 follows from properties and 2 and the orthogonality principle (HW exercise) EE 278B: Random Vectors 3 23
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2015 Timo Koski Matematisk statistik 24.09.2015 1 / 1 Learning outcomes Random vectors, mean vector, covariance matrix,
More informationSF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2014 Timo Koski () Mathematisk statistik 24.09.2014 1 / 75 Learning outcomes Random vectors, mean vector, covariance
More informationEigenvalues, Eigenvectors, Matrix Factoring, and Principal Components
Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they
More information6. Cholesky factorization
6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix
More informationInner Product Spaces and Orthogonality
Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,
More informationLecture Notes 1. Brief Review of Basic Probability
Probability Review Lecture Notes Brief Review of Basic Probability I assume you know basic probability. Chapters -3 are a review. I will assume you have read and understood Chapters -3. Here is a very
More informationCS229 Lecture notes. Andrew Ng
CS229 Lecture notes Andrew Ng Part X Factor analysis Whenwehavedatax (i) R n thatcomesfromamixtureofseveral Gaussians, the EM algorithm can be applied to fit a mixture model. In this setting, we usually
More informationDATA ANALYSIS II. Matrix Algorithms
DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where
More informationSections 2.11 and 5.8
Sections 211 and 58 Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1/25 Gesell data Let X be the age in in months a child speaks his/her first word and
More informationLinear Algebra Review. Vectors
Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length
More information15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
More informationFactorization Theorems
Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization
More informationLINEAR ALGEBRA. September 23, 2010
LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................
More informationInner Product Spaces
Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and
More informationThe Monte Carlo Framework, Examples from Finance and Generating Correlated Random Variables
Monte Carlo Simulation: IEOR E4703 Fall 2004 c 2004 by Martin Haugh The Monte Carlo Framework, Examples from Finance and Generating Correlated Random Variables 1 The Monte Carlo Framework Suppose we wish
More informationMath 115A HW4 Solutions University of California, Los Angeles. 5 2i 6 + 4i. (5 2i)7i (6 + 4i)( 3 + i) = 35i + 14 ( 22 6i) = 36 + 41i.
Math 5A HW4 Solutions September 5, 202 University of California, Los Angeles Problem 4..3b Calculate the determinant, 5 2i 6 + 4i 3 + i 7i Solution: The textbook s instructions give us, (5 2i)7i (6 + 4i)(
More information3. Let A and B be two n n orthogonal matrices. Then prove that AB and BA are both orthogonal matrices. Prove a similar result for unitary matrices.
Exercise 1 1. Let A be an n n orthogonal matrix. Then prove that (a) the rows of A form an orthonormal basis of R n. (b) the columns of A form an orthonormal basis of R n. (c) for any two vectors x,y R
More informationLINEAR ALGEBRA W W L CHEN
LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,
More informationNotes on Symmetric Matrices
CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.
More informationTorgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances
Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances It is possible to construct a matrix X of Cartesian coordinates of points in Euclidean space when we know the Euclidean
More informationLecture 8: Signal Detection and Noise Assumption
ECE 83 Fall Statistical Signal Processing instructor: R. Nowak, scribe: Feng Ju Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(, σ I n n and S = [s, s,...,
More informationNOTES ON LINEAR TRANSFORMATIONS
NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all
More informationSimilarity and Diagonalization. Similar Matrices
MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
More information1 Introduction to Matrices
1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns
More information[1] Diagonal factorization
8.03 LA.6: Diagonalization and Orthogonal Matrices [ Diagonal factorization [2 Solving systems of first order differential equations [3 Symmetric and Orthonormal Matrices [ Diagonal factorization Recall:
More informationSection 6.1 - Inner Products and Norms
Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,
More informationSome probability and statistics
Appendix A Some probability and statistics A Probabilities, random variables and their distribution We summarize a few of the basic concepts of random variables, usually denoted by capital letters, X,Y,
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
More informationNotes on Determinant
ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without
More informationReview Jeopardy. Blue vs. Orange. Review Jeopardy
Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round $200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?
More informationLectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain
Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain 1. Orthogonal matrices and orthonormal sets An n n real-valued matrix A is said to be an orthogonal
More informationIntroduction to Matrix Algebra
Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary
More informationOctober 3rd, 2012. Linear Algebra & Properties of the Covariance Matrix
Linear Algebra & Properties of the Covariance Matrix October 3rd, 2012 Estimation of r and C Let rn 1, rn, t..., rn T be the historical return rates on the n th asset. rn 1 rṇ 2 r n =. r T n n = 1, 2,...,
More informationLecture 1: Schur s Unitary Triangularization Theorem
Lecture 1: Schur s Unitary Triangularization Theorem This lecture introduces the notion of unitary equivalence and presents Schur s theorem and some of its consequences It roughly corresponds to Sections
More informationMULTIVARIATE PROBABILITY DISTRIBUTIONS
MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined
More informationUnderstanding and Applying Kalman Filtering
Understanding and Applying Kalman Filtering Lindsay Kleeman Department of Electrical and Computer Systems Engineering Monash University, Clayton 1 Introduction Objectives: 1. Provide a basic understanding
More informationQuadratic forms Cochran s theorem, degrees of freedom, and all that
Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us
More informationCONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation
Chapter 2 CONTROLLABILITY 2 Reachable Set and Controllability Suppose we have a linear system described by the state equation ẋ Ax + Bu (2) x() x Consider the following problem For a given vector x in
More information1 VECTOR SPACES AND SUBSPACES
1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such
More informationby the matrix A results in a vector which is a reflection of the given
Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that
More informationFactor analysis. Angela Montanari
Factor analysis Angela Montanari 1 Introduction Factor analysis is a statistical model that allows to explain the correlations between a large number of observed correlated variables through a small number
More informationThe Bivariate Normal Distribution
The Bivariate Normal Distribution This is Section 4.7 of the st edition (2002) of the book Introduction to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included
More informationMath 4310 Handout - Quotient Vector Spaces
Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable
More information5. Continuous Random Variables
5. Continuous Random Variables Continuous random variables can take any value in an interval. They are used to model physical characteristics such as time, length, position, etc. Examples (i) Let X be
More informationLinear Algebra Notes for Marsden and Tromba Vector Calculus
Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of
More informationMAT188H1S Lec0101 Burbulla
Winter 206 Linear Transformations A linear transformation T : R m R n is a function that takes vectors in R m to vectors in R n such that and T (u + v) T (u) + T (v) T (k v) k T (v), for all vectors u
More informationAbstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix multiplication).
MAT 2 (Badger, Spring 202) LU Factorization Selected Notes September 2, 202 Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix
More informationChapter 7. Permutation Groups
Chapter 7 Permutation Groups () We started the study of groups by considering planar isometries In the previous chapter, we learnt that finite groups of planar isometries can only be cyclic or dihedral
More informationIntroduction to General and Generalized Linear Models
Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby
More informationDirect Methods for Solving Linear Systems. Matrix Factorization
Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011
More informationIntroduction to Probability
Introduction to Probability EE 179, Lecture 15, Handout #24 Probability theory gives a mathematical characterization for experiments with random outcomes. coin toss life of lightbulb binary data sequence
More informationMATH 551 - APPLIED MATRIX THEORY
MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points
More informationLecture 5: Singular Value Decomposition SVD (1)
EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system
More informationLecture 21. The Multivariate Normal Distribution
Lecture. The Multivariate Normal Distribution. Definitions and Comments The joint moment-generating function of X,...,X n [also called the moment-generating function of the random vector (X,...,X n )]
More informationChapter 6. Orthogonality
6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be
More informationWhat is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference
0. 1. Introduction and probability review 1.1. What is Statistics? What is Statistics? Lecture 1. Introduction and probability review There are many definitions: I will use A set of principle and procedures
More informationNumerical Analysis Lecture Notes
Numerical Analysis Lecture Notes Peter J. Olver 6. Eigenvalues and Singular Values In this section, we collect together the basic facts about eigenvalues and eigenvectors. From a geometrical viewpoint,
More informationSOLVING LINEAR SYSTEMS
SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis
More informationVector and Matrix Norms
Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty
More information7 Gaussian Elimination and LU Factorization
7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method
More informationOverview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written
More informationSolving Linear Systems, Continued and The Inverse of a Matrix
, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing
More informationComponent Ordering in Independent Component Analysis Based on Data Power
Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals
More informationThe Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression
The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression The SVD is the most generally applicable of the orthogonal-diagonal-orthogonal type matrix decompositions Every
More informationSimilar matrices and Jordan form
Similar matrices and Jordan form We ve nearly covered the entire heart of linear algebra once we ve finished singular value decompositions we ll have seen all the most central topics. A T A is positive
More informationAu = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.
Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry
More informationA Tutorial on Probability Theory
Paola Sebastiani Department of Mathematics and Statistics University of Massachusetts at Amherst Corresponding Author: Paola Sebastiani. Department of Mathematics and Statistics, University of Massachusetts,
More informationOrthogonal Projections
Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors
More informationMultivariate Normal Distribution
Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues
More informationOverview of Monte Carlo Simulation, Probability Review and Introduction to Matlab
Monte Carlo Simulation: IEOR E4703 Fall 2004 c 2004 by Martin Haugh Overview of Monte Carlo Simulation, Probability Review and Introduction to Matlab 1 Overview of Monte Carlo Simulation 1.1 Why use simulation?
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete
More informationProbability Generating Functions
page 39 Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence
More information3. INNER PRODUCT SPACES
. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.
More informationInner products on R n, and more
Inner products on R n, and more Peyam Ryan Tabrizian Friday, April 12th, 2013 1 Introduction You might be wondering: Are there inner products on R n that are not the usual dot product x y = x 1 y 1 + +
More informationMultivariate normal distribution and testing for means (see MKB Ch 3)
Multivariate normal distribution and testing for means (see MKB Ch 3) Where are we going? 2 One-sample t-test (univariate).................................................. 3 Two-sample t-test (univariate).................................................
More informationSummary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)
Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume
More informationMaster s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
More information1 Sets and Set Notation.
LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most
More informationMath 431 An Introduction to Probability. Final Exam Solutions
Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <
More informationFor a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )
Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (1903-1987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll
More informationNotes on Orthogonal and Symmetric Matrices MENU, Winter 2013
Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,
More informationLeast-Squares Intersection of Lines
Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a
More informationBindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8
Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e
More informationLECTURE 4. Last time: Lecture outline
LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random
More informationThe Characteristic Polynomial
Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem
More informationLecture 2 Matrix Operations
Lecture 2 Matrix Operations transpose, sum & difference, scalar multiplication matrix multiplication, matrix-vector product matrix inverse 2 1 Matrix transpose transpose of m n matrix A, denoted A T or
More information3 Orthogonal Vectors and Matrices
3 Orthogonal Vectors and Matrices The linear algebra portion of this course focuses on three matrix factorizations: QR factorization, singular valued decomposition (SVD), and LU factorization The first
More informationProbability and Random Variables. Generation of random variables (r.v.)
Probability and Random Variables Method for generating random variables with a specified probability distribution function. Gaussian And Markov Processes Characterization of Stationary Random Process Linearly
More informationElasticity Theory Basics
G22.3033-002: Topics in Computer Graphics: Lecture #7 Geometric Modeling New York University Elasticity Theory Basics Lecture #7: 20 October 2003 Lecturer: Denis Zorin Scribe: Adrian Secord, Yotam Gingold
More information5. Orthogonal matrices
L Vandenberghe EE133A (Spring 2016) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal
More informationLinear Algebraic Equations, SVD, and the Pseudo-Inverse
Linear Algebraic Equations, SVD, and the Pseudo-Inverse Philip N. Sabes October, 21 1 A Little Background 1.1 Singular values and matrix inversion For non-smmetric matrices, the eigenvalues and singular
More informationChapter 4 Lecture Notes
Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a real-valued function defined on the sample space of some experiment. For instance,
More informationLinear Algebra: Determinants, Inverses, Rank
D Linear Algebra: Determinants, Inverses, Rank D 1 Appendix D: LINEAR ALGEBRA: DETERMINANTS, INVERSES, RANK TABLE OF CONTENTS Page D.1. Introduction D 3 D.2. Determinants D 3 D.2.1. Some Properties of
More informationA note on companion matrices
Linear Algebra and its Applications 372 (2003) 325 33 www.elsevier.com/locate/laa A note on companion matrices Miroslav Fiedler Academy of Sciences of the Czech Republic Institute of Computer Science Pod
More informationM2S1 Lecture Notes. G. A. Young http://www2.imperial.ac.uk/ ayoung
M2S1 Lecture Notes G. A. Young http://www2.imperial.ac.uk/ ayoung September 2011 ii Contents 1 DEFINITIONS, TERMINOLOGY, NOTATION 1 1.1 EVENTS AND THE SAMPLE SPACE......................... 1 1.1.1 OPERATIONS
More informationCovariance and Correlation
Covariance and Correlation ( c Robert J. Serfling Not for reproduction or distribution) We have seen how to summarize a data-based relative frequency distribution by measures of location and spread, such
More informationMehtap Ergüven Abstract of Ph.D. Dissertation for the degree of PhD of Engineering in Informatics
INTERNATIONAL BLACK SEA UNIVERSITY COMPUTER TECHNOLOGIES AND ENGINEERING FACULTY ELABORATION OF AN ALGORITHM OF DETECTING TESTS DIMENSIONALITY Mehtap Ergüven Abstract of Ph.D. Dissertation for the degree
More informationT ( a i x i ) = a i T (x i ).
Chapter 2 Defn 1. (p. 65) Let V and W be vector spaces (over F ). We call a function T : V W a linear transformation form V to W if, for all x, y V and c F, we have (a) T (x + y) = T (x) + T (y) and (b)
More informationCS3220 Lecture Notes: QR factorization and orthogonal transformations
CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss
More information