SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation


 Eugene Reeves
 2 years ago
 Views:
Transcription
1 SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas September 19, 2015
2 Outline Motivating Example 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
3 Outline Motivating Example 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
4 Motivation Motivating Example Until now we have seen how to fit distributions to data. The objective of Monte Carlo simulation is to generate data from distributions. Even if we have exact formulas for the distribution functions of individual random variables, it is not always possible (or easy) to generate distribution functions of their sum, or product, etc.
5 Motivation (Cont d) Motivating Example In applications such as supply chain management or project management, we often have available the distribution functions of the constituent parts of a large and complex system. Monte Carlo simulation allows us to generate samples for each constituent random variable, combine those to generate samples for the overall random variable, and then combine the samples of the overall random variable. These samples can then be used to estimate various quantities about the overall random variable, such as mean, variance, tail values, etc.
6 Toy Manufacturing Example X 1 X 3 Start Finish X 2 X 4 Parts simultaneously start at stations 1 and 2, then move to 3 and 4 respectively. When both stations 3 and 4 finish, the process is complete. Y = max{x 1 + X 3, X 2 + X 4 }.
7 Toy Manufacturing Example No. 2 X 1 X 3 Start Finish X 2 X 4 Parts simultaneously start at stations 1 and 2, then move to 3 and 4 respectively. When both stations 3 and 4 finish, the process is complete. Y 2 = max{max{x 1, X 2 } + X 3, X 2 + X 4 }.
8 General Approach Motivating Example From historical records we can generate cumulative distribution functions (cdfs) of the individual random variables X 1 through X 4. Even if we had formulas for the cdfs of the four random variabes X 1 through X 4, it would be extremely difficult to find a formula for the cdf of Y.
9 General Approach (Cont d) So instead we can generate lots of random samples of each of the four random variables X 1 through X 4, use these to compute lots of random samples of Y. We can use these samples to estimate various quantities, e.g., the mean and variance of Y. We can try to fit some distribution to these randomly generated samples, to get an approximate cdf of Y. We can fit an empirical distribution to the data, and estimate how close it is to the true distribution. Usually the middle bullet is not attempted.
10 Pertinent Questions Motivating Example Given cdfs of individual random variables X 1 through X 4, how do we generate samples of the random variables X 1 through X 4 with the specified distribution? How can we generate an empirical distribution of the random variabl Y? How well does this empirical distribution approximate the true but unknown distribution function?
11 Outline Motivating Example 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
12 Outline Motivating Example 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
13 Percentile Approach to Sampling Φ X x The grid points are uniformly spaced on the vertical axis, though not on the horizontal axis.
14 Generating Samples Using Uniform Distribution Suppose a cdf Φ X is specified. How can we generate samples of X with this distribution? Suppose Z is uniformly distributed on [0, 1], and let Φ X ( ) denote the distribution function of X. Then the r.v. Φ 1 X (Z) has the same distribution as X. To generate samples x 1,..., x n of X according to the distribution Φ X ( ), first generate samples z 1,..., z n with the uniform distribution, and then define x i = Φ 1 X (z i), i = 1,..., n.
15 Generating Samples Using Uniform Distribution (Cont d) The matlab command rand(n,m) generates an n m matrix of random (actually pseudorandom) numbers that are uniformly distributed. In particular, rand (n) generates m uniformly distributed random numbers. By substituting these numbers into Φ 1 X, we can generate the desired samples of X. Note that matlab provides inverse cdfs for many widely used distributions, such as Gaussian (normal), Poisson, etc. In addition, the function stblinv.m can be used to invert a given stable distribution, while triinv can be used to invert a triangular distribution.
16 Generating Samples Using Uniform Distribution (Cont d) If there are k independent random variables X 1,..., X k, we can generate a k n array of uniformly distributed (pseudo)random numbers Z by using the command Z = rand(k,n). Denote the entries of the k n matrix Z as z 11,..., z kn. Then we can generate n independent samples for each of the k random variables via x ij = Φ 1 X i (z ij ), i = 1,..., k, j = 1,..., n.
17 Outline Motivating Example 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
18 Monte Carlo Simulation Suppose Y = f(x 1,..., X k ) where X 1,..., X k are independent random variables. How can we generate samples of Y? Generate n independent samples each of the k random variables; call them x 11,..., x kn. Compute the samples y i = f(x 1i,..., x ki ), i = 1,..., n. Construct the empirical distribution function ˆΦ Y (u) = n I {u yi }. i=1
19 Toy Manufacturing Example Revisited Recall the example where Y = max{x 1 + X 3, X 2 + X 4 }. By substituting the samples into this formula, we can generate n independent samples of Y. So the question now arises: What do we do with these samples?
20 Empirical Distribution Function Suppose Y is the random variable of interest, and we have n independent samples of Y, call them y 1,..., y n. For each value of y, define the empirical distribution function ˆΦ Y (y) = 1 n n i=1 I yi y, where I denotes the indicator function it equals one if the statement in the subscript is true, and zero if it is false. So specifically ˆΦ Y (y) is just the fraction of the n samples that are smaller than or equal to y.
21 Empirical Distribution Function (Cont d) To construct the empirical distribution function, first sort all the samples y 1,..., y n in increasing order; call the result (y) 1,..., (y) n. Then construct a staircase function that jumps by 1/n at each sample (y) i. That is the empirical distribution function.
22 Depiction: Empirical Distribution ˆΦ Y (u) (y) 1 (y) 2 (y) 3 (y) 4 (y) 5 (y) 6 u
23 Theory Behind Monte Carlo Simulation Theory allows us to say just how many samples we need to draw, to get a desired level of accuracy of the estimate, with a given level of confidence. With confidence 1 δ it can be said that the true but unknown probability distribution function Φ Y (u) satisfies max u ˆΦ Y (u) Φ Y (u) θ, where θ(n, δ) = ( 1 2n log 2 ) 1/2. δ
24 Theory Behind Monte Carlo Simulation (Cont d) Turning around this inequality, if we want to approximate Φ Y (u) to accuracy θ with confidence 1 δ, then the minimum number of samples needed is n 1 2θ 2 log 2 δ. With this many samples, true but unknown probability distribution function Φ Y (u) lies within a band of width ɛ around the empirical probability distribution ˆΦ Y (u). For example, to approximate Φ Y ( ) to accuracy with confidence 95%, we require n 2951 samples. If we wish to be 99% sure, then we require 4, 239 samples.
25 Depiction: Error Bands for Empirical Distribution Thin green and vermilion staircase functions show upper and lower bounds for true but unknown distribution Φ Y ( ) These can be used to bound percentile values of Y with specified confidence and accuracy. ˆΦ Y (u) y i1 y i2 y i3 y i4 y i5 y i6 u
26 Estimating Value at Risk One of the most common uses of Monte Carlo simulation is estimating the Value at Risk (VaR). Suppose wish to determine a value V such that Pr{Y > V } α, where α is a prespecified level. Usual values of α are 0.01 or If α = 0.5, then V is the called the 95% Value at Risk, whereas if α = 0.01 then V is called the 99% Value at Risk.
27 Estimating Value at Risk (Cont d) We can express the VaR in terms of the complementary cdf: V = Φ 1 1 (1 α) = Φ (α). Y The difficulty however is that we don t know the true cdf Φ or the true ccdf Φ. This is where we can use the empirical distribution. Y
28 Estimating Value at Risk (Cont d) Suppose α = 0.05, so that we wish to estimate the 95% VaR. Choose θ = α/2 = Then choose the desired confidence level δ, and the corresponding number of samples n according to n 1 2θ 2 log 2 δ. With this many samples, we know that the empirical distribution function ˆΦ Y is within θ of the true but unknown distribution function.
29 Estimating Value at Risk (Cont d) Now compute ˆV according to ˆΦ Y ( ˆV ) 1 α/2 = 1 θ. Then, with confidence 1 δ, we can say that Φ Y ( ˆV ) 1 α. Therefore ˆV is an estimated VaR, at a confidence level of δ.
30 Estimating Value at Risk (Cont d) Often the VaR is estimated of some function of Y. For example, suppose Y is the time needed to complete some manufacturing job. The manufacturer receives a bonus for early completion and pays a penalty for late completion. We wish to estimate the VaR of the bonus/penalty. This situation can be modeled by defining the bonus B as a function of Y, with negative bonus corresponding to a penalty.
31 Estimating Value at Risk (Cont d) Once a level α is specified, we wish to estimate the value V B such that Pr{B(Y ) V B } = 1 α. Again, using the empirical distribution of Y, we can construct a corresponding empirical distribution of the bonus B, and use that to estimate the VaR of the bonus.
32 Estimating Value at Risk (Cont d) But often there is a simpler way to do this. If the bonus is a monotonic function of the time to completion (which is a reasonable assumption), then we simply compute (or estimate) the VaR of Y, and substitute that into the formula for the bonus.
33 Estimating Percentiles The VaR calculation applies to the far end of the distribution. The same philosophy can also be applied to estimating other percentiles, such the median for example. Suppose we wish to estimate the median value of Y. We have the empirical estimate ˆΦ Y, and we have chosen the number of samples n such that, with confidence 1 δ, we can assert that ˆΦ Y (u) Φ Y (u) θ u. Now the median corresponds to Φ 1 T (0.5). So we can compute ˆΦ 1 1 Y (0.5 θ) and ˆΦ Y (0.5 + θ). These numbers give a range for the median. To estimate other percentiles, just replace 0.5 by the desired number.
34 Hoeffding s Inequality If the random variable Y is bounded, then a very useful estimate known as Hoeffding s inequality becomes applicable. Note that if popular models such as Gaussian or lognormal distributions are used to model various quantities, then in principle the random variables are not bounded, and Hoeffding s inequality does not apply. But if triangular distributions (for example) are used, then Hoeffding s inequality does apply.
35 Hoeffding s Inequality (Cont d) Suppose Y is a random variable assuming values in a finite interval [a, b]. Suppose y 1,..., y n are independent samples of Y, and define ˆµ Y = 1 n n i=1 y i to be the empirical mean of Y. Let µ Y denote the true but unknown mean of Y. Hoeffding s inequality states that Pr{ ˆµ Y µ Y > ɛ} 2 exp( 2nɛ 2 /(b a) 2 ).
36 Hoeffding s Inequality (Cont d) Therefore, to estimate the quantity µ Y to within a specified accuracy ɛ with confidence 1 δ, we require ( ) (b a)2 2 n 2ɛ 2 log δ samples. We can also compute the accuracy ɛ in terms of the number of samples n and the confidence δ. [ ( )] b a 2 1/2 ɛ = 2n log. δ
37 Outline Motivating Example Example with Bounded Distributions Example with Unbounded Distributions 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
38 Outline Motivating Example Example with Bounded Distributions Example with Unbounded Distributions 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
39 Example with Bounded Distributions Example with Unbounded Distributions Specification of Individual Random Variables Suppose X 1 has a triangular distribution with minimum a 1 = 1, mode b 1 = 2, and maximum c 1 = 6. X 2 has a triangular distribution with minimum a 2 = 1, mode b 2 = 3, and maximum c 2 = 5. X 3 has a triangular distribution with minimum a 3 = 3, mode b 3 = 5, and maximum c 3 = 7. X 4 has a triangular distribution with minimum a 4 = 2, mode b 4 = 5, and maximum c 4 = 8.
40 Example with Bounded Distributions Example with Unbounded Distributions Determination of the Number of Samples Let us choose θ = 0.01, δ = This leads to n = 1 2θ 2 log 2 δ = Let us round this up to 26,500 samples. Repeating earlier steps leads to the empirical distribution shown in the next slide.
41 Example with Bounded Distributions Example with Unbounded Distributions Empirical Distribution of Processing Time 1 Empirical Distribution Function Phi hat(y) Values of Y
42 Estimating Median Processing Time Example with Bounded Distributions Example with Unbounded Distributions Following earlier steps, we find that [8.7431, ] to be the 99% confidence interval for the median processing time.
43 Estimating the Mean Processing Time Example with Bounded Distributions Example with Unbounded Distributions Because Y is now bounded, lying between 3 and 13, we can apply Hoeffding s inequality. Because we have 26,500 samples, we can compute the achievable accuracy at a confidence level of 1 δ using the formula [ ( )] b a 2 1/2 ɛ = 2n log. δ turns out to be ˆµ Y = Therefore we can assert with confidence 1 δ that the true mean µ(y ) lies in the interval [ˆµ Y ɛ, ˆµ Y + ɛ].
44 Example with Bounded Distributions Example with Unbounded Distributions Estimating the Mean Processing Time (Cont d) In the present case, choosing δ = 0.01 leads to ɛ = The empirical mean, that is, the average of the 26,500 samples of Y, In the present case, we can state with 99% confidence that the true mean of Y lies in the interval [8.7730, ]. This estimate does not differ too much from the estimate for the median, which is [8.7431, ]. This is because the empirical distribution of Y is not very skewed.
45 Outline Motivating Example Example with Bounded Distributions Example with Unbounded Distributions 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
46 Toy Manufacturing Example: Reprise Example with Bounded Distributions Example with Unbounded Distributions X 1 X 3 Start Finish X 2 X 4 Parts simultaneously start at stations 1 and 2, then move to 3 and 4 respectively. When both stations 3 and 4 finish, the process is complete. Y = max{x 1 + X 3, X 2 + X 4 }.
47 Example with Bounded Distributions Example with Unbounded Distributions Specification of Individual Random Variables Suppose X 1 is lognormally distributed with mean µ 1 = 1 and standard deviation (of log X 1 ) of s 1 = 0.5. X 2 has a triangular distribution with minimum a 2 = 1, mode b 2 = 2, and maximum c 2 = 6. X 3 is lognormally distributed with mean µ 1 = 2.25 and standard deviation (of log X 3 ) of s 1 = X 4 has a triangular distribution with minimum a 4 = 5, mode b 4 = 9, and maximum c 4 = 16.
48 Generation of Samples Example with Bounded Distributions Example with Unbounded Distributions Suppose we wish to approximate the distribution function of the total processing time Y to an accuracy of 0.025, with a confidence of Therefore θ = 0.025, δ = 0.01, which means that we require n = 1 2θ 2 log 2 δ = 4288 samples. We can round this up to n = 4300.
49 Generation of Samples (Cont d) Example with Bounded Distributions Example with Unbounded Distributions By using the appropriate matlab commands, we can generate n samples for each of the four random variables. By substituting into the expression Y = max{x 1 + X 3, X 2 + X 4 }. we can generate n independent samples of Y. This leads to the empirical distribution function shown in the next slide.
50 Example with Bounded Distributions Example with Unbounded Distributions Empirical Distribution of Processing Time 1 Empirical Distribution Function via Monte Carlo Simulation Empirical Distribution Function of Y Values of Y1
51 Example with Bounded Distributions Example with Unbounded Distributions Estimating Value at Risk of Processing Time We have chosen θ = So we can estimate the 1 θ Value at Risk (97.5% VaR) of Y using the empirical distribution. Note that 1 2θ = 095. So, with confidence 1 δ = 0.99, we can say that the 95% VaR of the empirical distribution of Y is no larger than the 97.5% VaR of the true but unknown distribution of Y. This value turns out to be Therefore we are 99% sure that the 97.5% VaR of Y is not larger than this number.
52 Estimating the Median Example with Bounded Distributions Example with Unbounded Distributions We would like to estimate the median value of Y, which is Φ 1 Y (0.5). By finding the range of values [ˆΦ 1 1 Y (0.5 θ), ˆΦ Y (0.5 + θ)], we can get an estimate for the mediam value of Y, with confidence 1 δ. This interval turns out to be [ , ]. So we are 99% sure that the median value of Y lies in this interval. Because the lognormal distribution is unbounded, we cannot apply Hoeffding s inequality to this problem.
53 Outline Motivating Example Definition and Characterization of Independence Covariance and Correlation Coefficient 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
54 Outline Motivating Example Definition and Characterization of Independence Covariance and Correlation Coefficient 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
55 Independence of Real Random Variables Definition and Characterization of Independence Covariance and Correlation Coefficient There are two equivalent ways of defining independence in this case. X, Y are independent if or equivalently Φ X,Y (a, b) = Φ X (a) Φ Y (b) x, y, φ X,Y (x, y) = φ X (x) φ Y (y) x, y.
56 Sums of Independent Random Variables Definition and Characterization of Independence Covariance and Correlation Coefficient Suppose X, Y are independent r.v.s with densities φ X and φ Y respectively. Then the r.v. Z = X + Y has the density φ Z (z) = = z z φ X (u)φ Y (z u)du φ X (z v)φ Y (v)dv. In other words, the density of X + Y is the convolution of the densities of X and Y. If X and Y are not indepedent, then this statement is false in general.
57 Outline Motivating Example Definition and Characterization of Independence Covariance and Correlation Coefficient 1 Motivating Example 2 3 Example with Bounded Distributions Example with Unbounded Distributions 4 Definition and Characterization of Independence Covariance and Correlation Coefficient
58 Covariance Motivating Example Definition and Characterization of Independence Covariance and Correlation Coefficient Suppose X, Y are realvalued r.v.s. Define the expected values E[X], E[Y ], variances V (X), V (Y ), and standard deviations σ(x) = (V (X)) 1/2 and σ(y ) = (V (Y )) 1/2. The quantity C(X, Y ) = E[(X E(X))(Y E(Y ))] = E[XY ] E[X]E[Y ] is called the covariance of X and Y.
59 Correlation Coefficient Definition and Characterization of Independence Covariance and Correlation Coefficient ρ(x, Y ) = C(X, Y ) E[XY ] E[X]E[Y ] = σ(x)σ(y ) σ(x)σ(y ) is called the correlation coefficient between X and Y. ρ(x, Y ) is always in the interval [ 1, 1]. If ρ(x, Y ) > 0 we say that X, Y are positively correlated; if ρ(x, Y ) < 0 we say that X, Y are negatively correlated; and if ρ(x, Y ) = 0 we say that X, Y are uncorrelated. Note that the correlation coefficient is invariant under linear transformations. So if a, b, c, d are real numbers, then ρ(x, Y ) = ρ(ax + b, cy + d).
60 Correlation Coefficient 2 Definition and Characterization of Independence Covariance and Correlation Coefficient Common Misinterpretation: If X, Y are uncorrelated, then they are independent. The correlation coefficient ρ(x, Y ) tells only whether E[XY ] is more or less than the product E[X]E[Y ]. Fact: If X, Y are independent, then E(XY ) = E(X) E(Y ). Therefore ρ(x, Y ) = 0 if X, Y are independent. But the converse is not true at all!
61 More Than Two Random Variables Definition and Characterization of Independence Covariance and Correlation Coefficient We will discuss the case where there are, not just two, but d 2 realvalued random variables X 1,..., X d. In this case we can define an d d covariance matrix C defined by c ij = E[X i X j ] E[X i ]E[X j ], i, j = 1,..., d. Then C is a symmetric and positive semidefinite matrix, that is, all of its eigenvalues are real and nonnegative.
62 Multivariate Gaussian Distribution Definition and Characterization of Independence Covariance and Correlation Coefficient Suppose d 2 is some integer, µ R d, and Σ is a d d symmetric and positive definite matrix. Then the ddimensional joint density function φ X (x) = 1 (2π) d/2 det(σ) 1/2 exp( (1/2)(x µ)t Σ 1 (x µ)) defines the ddimensional Gaussian distribution with mean µ and covariance matrix Σ. It is easy to check that it is a generalization of the onedimensional Gaussian density function φ(x) = 1 2πσ exp( (x µ) 2 /2σ 2 ).
63 Multivariate Gaussian Distribution 2 Definition and Characterization of Independence Covariance and Correlation Coefficient The ddimensional Gaussian distribution defines a collection of d random variables with the properties that and covariance matrix Σ. E(X) = µ, that is, E(X i ) = µ i, Important Property: It is easy to see that the d random variables are pairwise uncorrelated if and only if the matrix Σ is diagonal. However, for Gaussian distributions only, it can be shown that if Σ is diagonal, then the d random variables are also pairwise independent.
64 Simulating Correlated Gaussian Variables Definition and Characterization of Independence Covariance and Correlation Coefficient The Matlab command norminv can be used with scalars as well as matrices. Thus if x is an ndimensional vector consisting of samples generated using the uniform distribution, then y = norminv(x, µ, σ) generates Gaussian samples with mean µ and standard deviation σ. If X is an n d matrix consisting of independent samples generated using the uniform distribution, µ is a ddimensional vector, and Σ is a d d matrix, then y = norminv(x, µ, Σ) generates Gaussian samples with mean µ and covariance matrix Σ.
Notes for STA 437/1005 Methods for Multivariate Data
Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.
More informationTopic 4: Multivariate random variables. Multiple random variables
Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly
More informationDepartment of Mathematics, Indian Institute of Technology, Kharagpur Assignment 23, Probability and Statistics, March 2015. Due:March 25, 2015.
Department of Mathematics, Indian Institute of Technology, Kharagpur Assignment 3, Probability and Statistics, March 05. Due:March 5, 05.. Show that the function 0 for x < x+ F (x) = 4 for x < for x
More informationChapter 3 RANDOM VARIATE GENERATION
Chapter 3 RANDOM VARIATE GENERATION In order to do a Monte Carlo simulation either by hand or by computer, techniques must be developed for generating values of random variables having known distributions.
More informationP (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )
Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =
More informationCovariance and Correlation. Consider the joint probability distribution f XY (x, y).
Chapter 5: JOINT PROBABILITY DISTRIBUTIONS Part 2: Section 52 Covariance and Correlation Consider the joint probability distribution f XY (x, y). Is there a relationship between X and Y? If so, what kind?
More information4. Joint Distributions of Two Random Variables
4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint
More informationSome probability and statistics
Appendix A Some probability and statistics A Probabilities, random variables and their distribution We summarize a few of the basic concepts of random variables, usually denoted by capital letters, X,Y,
More informationGenerating Random Numbers Variance Reduction QuasiMonte Carlo. Simulation Methods. Leonid Kogan. MIT, Sloan. 15.450, Fall 2010
Simulation Methods Leonid Kogan MIT, Sloan 15.450, Fall 2010 c Leonid Kogan ( MIT, Sloan ) Simulation Methods 15.450, Fall 2010 1 / 35 Outline 1 Generating Random Numbers 2 Variance Reduction 3 QuasiMonte
More informationSF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2015 Timo Koski Matematisk statistik 24.09.2015 1 / 1 Learning outcomes Random vectors, mean vector, covariance matrix,
More informationOutline. Random Variables. Examples. Random Variable
Outline Random Variables M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno Random variables. CDF and pdf. Joint random variables. Correlated, independent, orthogonal. Correlation,
More informationMaximum Likelihood Estimation
Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for
More informationWorked examples Multiple Random Variables
Worked eamples Multiple Random Variables Eample Let X and Y be random variables that take on values from the set,, } (a) Find a joint probability mass assignment for which X and Y are independent, and
More informationJoint Exam 1/P Sample Exam 1
Joint Exam 1/P Sample Exam 1 Take this practice exam under strict exam conditions: Set a timer for 3 hours; Do not stop the timer for restroom breaks; Do not look at your notes. If you believe a question
More informationProbability and Statistics
CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b  0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute  Systems and Modeling GIGA  Bioinformatics ULg kristel.vansteen@ulg.ac.be
More informationFor a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )
Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (19031987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll
More informationChapters 5. Multivariate Probability Distributions
Chapters 5. Multivariate Probability Distributions Random vectors are collection of random variables defined on the same sample space. Whenever a collection of random variables are mentioned, they are
More informationSummary of Formulas and Concepts. Descriptive Statistics (Ch. 14)
Summary of Formulas and Concepts Descriptive Statistics (Ch. 14) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume
More informationEconometrics Simple Linear Regression
Econometrics Simple Linear Regression Burcu Eke UC3M Linear equations with one variable Recall what a linear equation is: y = b 0 + b 1 x is a linear equation with one variable, or equivalently, a straight
More informationRandom Variables, Expectation, Distributions
Random Variables, Expectation, Distributions CS 5960/6960: Nonparametric Methods Tom Fletcher January 21, 2009 Review Random Variables Definition A random variable is a function defined on a probability
More informationSF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2014 Timo Koski () Mathematisk statistik 24.09.2014 1 / 75 Learning outcomes Random vectors, mean vector, covariance
More informationThe Monte Carlo Framework, Examples from Finance and Generating Correlated Random Variables
Monte Carlo Simulation: IEOR E4703 Fall 2004 c 2004 by Martin Haugh The Monte Carlo Framework, Examples from Finance and Generating Correlated Random Variables 1 The Monte Carlo Framework Suppose we wish
More informationErrata and updates for ASM Exam C/Exam 4 Manual (Sixteenth Edition) sorted by page
Errata for ASM Exam C/4 Study Manual (Sixteenth Edition) Sorted by Page 1 Errata and updates for ASM Exam C/Exam 4 Manual (Sixteenth Edition) sorted by page Practice exam 1:9, 1:22, 1:29, 9:5, and 10:8
More informationUsing pivots to construct confidence intervals. In Example 41 we used the fact that
Using pivots to construct confidence intervals In Example 41 we used the fact that Q( X, µ) = X µ σ/ n N(0, 1) for all µ. We then said Q( X, µ) z α/2 with probability 1 α, and converted this into a statement
More informationOverview of Monte Carlo Simulation, Probability Review and Introduction to Matlab
Monte Carlo Simulation: IEOR E4703 Fall 2004 c 2004 by Martin Haugh Overview of Monte Carlo Simulation, Probability Review and Introduction to Matlab 1 Overview of Monte Carlo Simulation 1.1 Why use simulation?
More informationProbability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur
Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special DistributionsVI Today, I am going to introduce
More information1 Sufficient statistics
1 Sufficient statistics A statistic is a function T = rx 1, X 2,, X n of the random sample X 1, X 2,, X n. Examples are X n = 1 n s 2 = = X i, 1 n 1 the sample mean X i X n 2, the sample variance T 1 =
More informationMath 141. Lecture 7: Variance, Covariance, and Sums. Albyn Jones 1. 1 Library 304. jones/courses/141
Math 141 Lecture 7: Variance, Covariance, and Sums Albyn Jones 1 1 Library 304 jones@reed.edu www.people.reed.edu/ jones/courses/141 Last Time Variance: expected squared deviation from the mean: Standard
More informationMath 431 An Introduction to Probability. Final Exam Solutions
Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <
More informationMATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators...
MATH4427 Notebook 2 Spring 2016 prepared by Professor Jenny Baglivo c Copyright 20092016 by Jenny A. Baglivo. All Rights Reserved. Contents 2 MATH4427 Notebook 2 3 2.1 Definitions and Examples...................................
More informationMT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...
MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 20042012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................
More informationOctober 3rd, 2012. Linear Algebra & Properties of the Covariance Matrix
Linear Algebra & Properties of the Covariance Matrix October 3rd, 2012 Estimation of r and C Let rn 1, rn, t..., rn T be the historical return rates on the n th asset. rn 1 rṇ 2 r n =. r T n n = 1, 2,...,
More informationVariances and covariances
Chapter 4 Variances and covariances 4.1 Overview The expected value of a random variable gives a crude measure for the center of location of the distribution of that random variable. For instance, if the
More informationCovariance and Correlation
Covariance and Correlation ( c Robert J. Serfling Not for reproduction or distribution) We have seen how to summarize a databased relative frequency distribution by measures of location and spread, such
More informationDongfeng Li. Autumn 2010
Autumn 2010 Chapter Contents Some statistics background; ; Comparing means and proportions; variance. Students should master the basic concepts, descriptive statistics measures and graphs, basic hypothesis
More informationLecture 6: Discrete & Continuous Probability and Random Variables
Lecture 6: Discrete & Continuous Probability and Random Variables D. Alex Hughes Math Camp September 17, 2015 D. Alex Hughes (Math Camp) Lecture 6: Discrete & Continuous Probability and Random September
More informationExact Confidence Intervals
Math 541: Statistical Theory II Instructor: Songfeng Zheng Exact Confidence Intervals Confidence intervals provide an alternative to using an estimator ˆθ when we wish to estimate an unknown parameter
More informationIntroduction to General and Generalized Linear Models
Introduction to General and Generalized Linear Models General Linear Models  part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK2800 Kgs. Lyngby
More information5. Continuous Random Variables
5. Continuous Random Variables Continuous random variables can take any value in an interval. They are used to model physical characteristics such as time, length, position, etc. Examples (i) Let X be
More informationRandom Vectors and the Variance Covariance Matrix
Random Vectors and the Variance Covariance Matrix Definition 1. A random vector X is a vector (X 1, X 2,..., X p ) of jointly distributed random variables. As is customary in linear algebra, we will write
More information3.6: General Hypothesis Tests
3.6: General Hypothesis Tests The χ 2 goodness of fit tests which we introduced in the previous section were an example of a hypothesis test. In this section we now consider hypothesis tests more generally.
More informationDefinition 6.1.1. A r.v. X has a normal distribution with mean µ and variance σ 2, where µ R, and σ > 0, if its density is f(x) = 1. 2σ 2.
Chapter 6 Brownian Motion 6. Normal Distribution Definition 6... A r.v. X has a normal distribution with mean µ and variance σ, where µ R, and σ > 0, if its density is fx = πσ e x µ σ. The previous definition
More informationTHE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE. Alexander Barvinok
THE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE Alexer Barvinok Papers are available at http://www.math.lsa.umich.edu/ barvinok/papers.html This is a joint work with J.A. Hartigan
More information15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
More informationGaussian Conjugate Prior Cheat Sheet
Gaussian Conjugate Prior Cheat Sheet Tom SF Haines 1 Purpose This document contains notes on how to handle the multivariate Gaussian 1 in a Bayesian setting. It focuses on the conjugate prior, its Bayesian
More informationEfficiency and the CramérRao Inequality
Chapter Efficiency and the CramérRao Inequality Clearly we would like an unbiased estimator ˆφ (X of φ (θ to produce, in the long run, estimates which are fairly concentrated i.e. have high precision.
More informationThe Bivariate Normal Distribution
The Bivariate Normal Distribution This is Section 4.7 of the st edition (2002) of the book Introduction to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included
More informationMaster s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
More informationDefinition The covariance of X and Y, denoted by cov(x, Y ) is defined by. cov(x, Y ) = E(X µ 1 )(Y µ 2 ).
Correlation Regression Bivariate Normal Suppose that X and Y are r.v. s with joint density f(x y) and suppose that the means of X and Y are respectively µ 1 µ 2 and the variances are 1 2. Definition The
More informationBasics Inversion and related concepts Random vectors Matrix calculus. Matrix algebra. Patrick Breheny. January 20
Matrix algebra January 20 Introduction Basics The mathematics of multiple regression revolves around ordering and keeping track of large arrays of numbers and solving systems of equations The mathematical
More informationBasics of Statistical Machine Learning
CS761 Spring 2013 Advanced Machine Learning Basics of Statistical Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu Modern machine learning is rooted in statistics. You will find many familiar
More informationMULTIVARIATE PROBABILITY DISTRIBUTIONS
MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined
More informationST 371 (VIII): Theory of Joint Distributions
ST 371 (VIII): Theory of Joint Distributions So far we have focused on probability distributions for single random variables. However, we are often interested in probability statements concerning two or
More informationPermutation Tests for Comparing Two Populations
Permutation Tests for Comparing Two Populations Ferry Butar Butar, Ph.D. JaeWan Park Abstract Permutation tests for comparing two populations could be widely used in practice because of flexibility of
More informationAggregate Loss Models
Aggregate Loss Models Chapter 9 Stat 477  Loss Models Chapter 9 (Stat 477) Aggregate Loss Models Brian Hartman  BYU 1 / 22 Objectives Objectives Individual risk model Collective risk model Computing
More informationL10: Probability, statistics, and estimation theory
L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian
More informationNotes on Continuous Random Variables
Notes on Continuous Random Variables Continuous random variables are random quantities that are measured on a continuous scale. They can usually take on any value over some interval, which distinguishes
More information3. The Multivariate Normal Distribution
3. The Multivariate Normal Distribution 3.1 Introduction A generalization of the familiar bell shaped normal density to several dimensions plays a fundamental role in multivariate analysis While real data
More information4. Continuous Random Variables, the Pareto and Normal Distributions
4. Continuous Random Variables, the Pareto and Normal Distributions A continuous random variable X can take any value in a given range (e.g. height, weight, age). The distribution of a continuous random
More informationMATH 201. Final ANSWERS August 12, 2016
MATH 01 Final ANSWERS August 1, 016 Part A 1. 17 points) A bag contains three different types of dice: four 6sided dice, five 8sided dice, and six 0sided dice. A die is drawn from the bag and then rolled.
More informationSYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis
SYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu October 17, 2015 Outline
More informationUNIT 2 MATRICES  I 2.0 INTRODUCTION. Structure
UNIT 2 MATRICES  I Matrices  I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress
More informationProbability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0
Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0 This primer provides an overview of basic concepts and definitions in probability and statistics. We shall
More information, for x = 0, 1, 2, 3,... (4.1) (1 + 1/n) n = 2.71828... b x /x! = e b, x=0
Chapter 4 The Poisson Distribution 4.1 The Fish Distribution? The Poisson distribution is named after SimeonDenis Poisson (1781 1840). In addition, poisson is French for fish. In this chapter we will
More information6. Distribution and Quantile Functions
Virtual Laboratories > 2. Distributions > 1 2 3 4 5 6 7 8 6. Distribution and Quantile Functions As usual, our starting point is a random experiment with probability measure P on an underlying sample spac
More informationQuadratic forms Cochran s theorem, degrees of freedom, and all that
Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us
More informationVariance of OLS Estimators and Hypothesis Testing. Randomness in the model. GM assumptions. Notes. Notes. Notes. Charlie Gibbons ARE 212.
Variance of OLS Estimators and Hypothesis Testing Charlie Gibbons ARE 212 Spring 2011 Randomness in the model Considering the model what is random? Y = X β + ɛ, β is a parameter and not random, X may be
More informationAdvanced 3G and 4G Wireless Communication Prof. Aditya K. Jagannatham Department of Electrical Engineering Indian Institute of Technology, Kanpur
Advanced 3G and 4G Wireless Communication Prof. Aditya K. Jagannatham Department of Electrical Engineering Indian Institute of Technology, Kanpur Lecture  3 Rayleigh Fading and BER of Wired Communication
More informationUNIT I: RANDOM VARIABLES PART A TWO MARKS
UNIT I: RANDOM VARIABLES PART A TWO MARKS 1. Given the probability density function of a continuous random variable X as follows f(x) = 6x (1x) 0
More informationCorrelation in Random Variables
Correlation in Random Variables Lecture 11 Spring 2002 Correlation in Random Variables Suppose that an experiment produces two random variables, X and Y. What can we say about the relationship between
More informationJoint Probability Distributions and Random Samples. Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage
5 Joint Probability Distributions and Random Samples Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage Two Discrete Random Variables The probability mass function (pmf) of a single
More informationECE302 Spring 2006 HW7 Solutions March 11, 2006 1
ECE32 Spring 26 HW7 Solutions March, 26 Solutions to HW7 Note: Most of these solutions were generated by R. D. Yates and D. J. Goodman, the authors of our textbook. I have added comments in italics where
More information1. χ 2 minimization 2. Fits in case of of systematic errors
Data fitting Volker Blobel University of Hamburg March 2005 1. χ 2 minimization 2. Fits in case of of systematic errors Keys during display: enter = next page; = next page; = previous page; home = first
More informationProbability and Random Variables. Generation of random variables (r.v.)
Probability and Random Variables Method for generating random variables with a specified probability distribution function. Gaussian And Markov Processes Characterization of Stationary Random Process Linearly
More informationMonte Carlo Simulation
1 Monte Carlo Simulation Stefan Weber Leibniz Universität Hannover email: sweber@stochastik.unihannover.de web: www.stochastik.unihannover.de/ sweber Monte Carlo Simulation 2 Quantifying and Hedging
More informationVariance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers
Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications
More informationMath 576: Quantitative Risk Management
Math 576: Quantitative Risk Management Haijun Li lih@math.wsu.edu Department of Mathematics Washington State University Week 4 Haijun Li Math 576: Quantitative Risk Management Week 4 1 / 22 Outline 1 Basics
More informationVector and Matrix Norms
Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a nonempty
More informationDiscrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 18. A Brief Introduction to Continuous Probability
CS 7 Discrete Mathematics and Probability Theory Fall 29 Satish Rao, David Tse Note 8 A Brief Introduction to Continuous Probability Up to now we have focused exclusively on discrete probability spaces
More informationNumerical Summarization of Data OPRE 6301
Numerical Summarization of Data OPRE 6301 Motivation... In the previous session, we used graphical techniques to describe data. For example: While this histogram provides useful insight, other interesting
More informationProbability Theory. Elementary rules of probability Sum rule. Product rule. p. 23
Probability Theory Uncertainty is key concept in machine learning. Probability provides consistent framework for the quantification and manipulation of uncertainty. Probability of an event is the fraction
More informationDescriptive Statistics
Y520 Robert S Michael Goal: Learn to calculate indicators and construct graphs that summarize and describe a large quantity of values. Using the textbook readings and other resources listed on the web
More informationMultivariate normal distribution and testing for means (see MKB Ch 3)
Multivariate normal distribution and testing for means (see MKB Ch 3) Where are we going? 2 Onesample ttest (univariate).................................................. 3 Twosample ttest (univariate).................................................
More informationModule 3: Correlation and Covariance
Using Statistical Data to Make Decisions Module 3: Correlation and Covariance Tom Ilvento Dr. Mugdim Pašiƒ University of Delaware Sarajevo Graduate School of Business O ften our interest in data analysis
More informationSpatial Statistics Chapter 3 Basics of areal data and areal data modeling
Spatial Statistics Chapter 3 Basics of areal data and areal data modeling Recall areal data also known as lattice data are data Y (s), s D where D is a discrete index set. This usually corresponds to data
More information( ) = P Z > = P( Z > 1) = 1 Φ(1) = 1 0.8413 = 0.1587 P X > 17
4.6 I company that manufactures and bottles of apple juice uses a machine that automatically fills 6 ounce bottles. There is some variation, however, in the amounts of liquid dispensed into the bottles
More informationThe sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].
Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real
More informationChapter 4  Lecture 1 Probability Density Functions and Cumul. Distribution Functions
Chapter 4  Lecture 1 Probability Density Functions and Cumulative Distribution Functions October 21st, 2009 Review Probability distribution function Useful results Relationship between the pdf and the
More information3 Random vectors and multivariate normal distribution
3 Random vectors and multivariate normal distribution As we saw in Chapter 1, a natural way to think about repeated measurement data is as a series of random vectors, one vector corresponding to each unit.
More informationSimple Linear Regression Inference
Simple Linear Regression Inference 1 Inference requirements The Normality assumption of the stochastic term e is needed for inference even if it is not a OLS requirement. Therefore we have: Interpretation
More informationLecture 8: More Continuous Random Variables
Lecture 8: More Continuous Random Variables 26 September 2005 Last time: the eponential. Going from saying the density e λ, to f() λe λ, to the CDF F () e λ. Pictures of the pdf and CDF. Today: the Gaussian
More informationHypothesis Testing COMP 245 STATISTICS. Dr N A Heard. 1 Hypothesis Testing 2 1.1 Introduction... 2 1.2 Error Rates and Power of a Test...
Hypothesis Testing COMP 45 STATISTICS Dr N A Heard Contents 1 Hypothesis Testing 1.1 Introduction........................................ 1. Error Rates and Power of a Test.............................
More informationStatistical Foundations: Measures of Location and Central Tendency and Summation and Expectation
Statistical Foundations: and Central Tendency and and Lecture 4 September 5, 2006 Psychology 790 Lecture #49/05/2006 Slide 1 of 26 Today s Lecture Today s Lecture Where this Fits central tendency/location
More informationElliptical copulae. Dorota Kurowicka, Jolanta Misiewicz, Roger Cooke
Elliptical copulae Dorota Kurowicka, Jolanta Misiewicz, Roger Cooke Abstract: In this paper we construct a copula, that is, a distribution with uniform marginals. This copula is continuous and can realize
More informationPoint and Interval Estimates
Point and Interval Estimates Suppose we want to estimate a parameter, such as p or µ, based on a finite sample of data. There are two main methods: 1. Point estimate: Summarize the sample by a single number
More information6.4 Normal Distribution
Contents 6.4 Normal Distribution....................... 381 6.4.1 Characteristics of the Normal Distribution....... 381 6.4.2 The Standardized Normal Distribution......... 385 6.4.3 Meaning of Areas under
More informationGeneralized Linear Models. Today: definition of GLM, maximum likelihood estimation. Involves choice of a link function (systematic component)
Generalized Linear Models Last time: definition of exponential family, derivation of mean and variance (memorize) Today: definition of GLM, maximum likelihood estimation Include predictors x i through
More informationExercises with solutions (1)
Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal
More information1 Inner Products and Norms on Real Vector Spaces
Math 373: Principles Techniques of Applied Mathematics Spring 29 The 2 Inner Product 1 Inner Products Norms on Real Vector Spaces Recall that an inner product on a real vector space V is a function from
More informationWe call this set an ndimensional parallelogram (with one vertex 0). We also refer to the vectors x 1,..., x n as the edges of P.
Volumes of parallelograms 1 Chapter 8 Volumes of parallelograms In the present short chapter we are going to discuss the elementary geometrical objects which we call parallelograms. These are going to
More information