1 Review of Probability

Transcription

1 Copyright c 27 by Karl Sigma 1 Review of Probability Radom variables are deoted by X, Y, Z, etc. The cumulative distributio fuctio (c.d.f.) of a radom variable X is deoted by F (x) = P (X x), < x <, ad if the radom variable is cotiuous the its probability desity fuctio is deoted by f(x) which is related to F (x) via f(x) = F (x) = d dx F (x) F (x) = x f(y)dy. The probability mass fuctio (p.m.f.) of a discrete radom variable is give by p(k) = P (X = k), < k <, for itegers k. 1 F (x) = P (X > x) is called the tail of X ad is deoted by F (x) = 1 F (x). Whereas F (x) icreases to 1 as x, ad decreases to as x, the tail F (x) decreases to as x ad icreases to 1 as x. If a r.v. X has a certai distributio with c.d.f. F (x) = P (X x), the we write, for simplicity of expressio, X F. (1) 1.1 Momets ad variace The expected value of a r.v. is deote by E(X) ad defied by E(X) = E(X) = k= kp(k), discrete case, xf(x)dx, cotiuous case. E(X) is also referred to as the first momet or mea of X (or of its distributio). Higher momets E(X ), 1 ca be computed via E(X ) = E(X ) = k= k p(k), discrete case, x f(x)dx, cotiuous case, ad more geerally E(g(X)) for a fuctio g = g(x) ca be computed via E(g(X)) = E(g(X)) = k= g(k)p(k), discrete case, g(x)f(x)dx, cotiuous case. 1

2 (Letig g(x) = x yields momets for example.) Fially, the variace of X is deoted by V ar(x), defied by E{ X E(X) 2 }, ad ca be computed via V ar(x) = E(X 2 ) E 2 (X), (2) the secod momet mius the square of the first momet. We usually deote the variace by σ 2 = V ar(x) ad whe ecessary (to avoid cofusio) iclude X as a subscript, σ 2 X = V ar(x). σ = V ar(x) is called the stadard deviatio of X. For ay r.v. X ad ay umber a where For ay two r.v.s. X ad Y If X ad Y are idepedet, the E(aX) = ae(x), ad V ar(ax) = a 2 V ar(x). (3) E(X + Y ) = E(X) + E(Y ). (4) V ar(x + Y ) = V ar(x) + V ar(y ). (5) The above properties geeralize i the obvious fashio to to ay fiite umber of r.v.s. I geeral (idepedet or ot) V ar(x + Y ) = V ar(x) + V (Y ) + 2Cov(X, Y ), Cov(X, Y ) def = E(XY ) E(X)E(Y ), is called the covariace betwee X ad Y, ad is usually deoted by σ X,Y = Cov(X, Y ). Whe Cov(X, Y ) >, X ad Y are said to be positively correlated, whereas whe Cov(X, Y ) <, X ad Y are said to be egatively correlated. Whe Cov(X, Y ) =, X ad Y are said to be ucorrelated, ad i geeral this is weaker tha idepedece of X ad Y : there are examples of ucorrelated r.v.s. that are ot idepedet. Note i passig that Cov(X, X) = V ar(x). The correlatio coefficiet of X, Y is defied by ρ = σ X,Y σ X σ Y, ad it always holds that 1 ρ 1. Whe ρ = 1, X ad Y are said to be perfectly (positively) correlated. 1.2 Momet geeratig fuctios The momet geeratig fuctio (mgf) of a r.v. s (, ) by X (or its distributio) is defied for all M(s) def = E(e sx ) (6) ( ) = e sx f(x)dx = e sk p(k) i the discrete r.v. case It is so called because it geerates the momets of X by differetiatio at s = : M () = E(X), (7) 2

3 ad more geerally M () () = E(X ), 1. (8) The mgf uiquely determies a distributio i that o two distributios ca have the same mgf. So kowig a mgf characterizes the distributio i questio. If X ad Y are idepedet, the E(e s(x+y ) ) = E(e sx e sy ) = E(e sx )E(e sy ), ad we coclude that the mgf of a idepedet sum is the product of the idividual mgf s. Sometimes to stress the particular r.v. X, we write M X (s). The the above idepedece property ca be cocisely expressed as M X+Y (s) = M X (s)m Y (s), whe X ad Y are idepedet. Remark 1.1 For a give distributio, M(s) = is possible for some values of s, but there is a large useful class of distributios for which M(s) < for all s i a eighborhood of the origi, that is, for s ( ɛ, ɛ) with ɛ > suffietly small. Such distributios are referred to as light-tailed because their tails ca be show to ted to zero quickly. There also exists distributios for which o such eighborhood exists ad this ca be so eve if the distributio has fiite momets of all orders (see the logormal distributio for example). A large class of such distributios are referred to as heavy-tailed because their tails ted to zero slowly. Remark 1.2 For o-egative r.v.s. X, it is sometimes more commo to use the Laplace trasform, L(s) = E(e sx ), s, which is always fiite, ad the ( 1) L () () = E(X ), 1. For discrete r.v.s. X, it is sometimes more commo to use M(z) = E(z X ) = k= z k p(k), z 1 for the mgf i which case momets ca be geerated via M (1) = E(X), M (1) = E((X)(X 1)), M () (1) = E(X(X 1) (X ( 1))), Examples of well-kow distributios Discrete case 1. Beroulli distributio with success probability p: With < p < 1 a costat, X has p.m.f. p(k) = P (X = k) give by p(1) = p, p() = 1 p, p(k) =, otherwise. Thus X oly takes o the values 1 (success) or (failure). A simple computatio yields E(X) = p V ar(x) = p(1 p) M(s) = pe s + 1 p. Beroulli r.v.s. arise aturally as the idicator fuctio, X = I{A}, of a evet A, where { I{A} def 1, if the evet A occurs; =, otherwise. 3

4 The p = P (X = 1) = P (A) is the probability that the evet A occurs. For example, if you flip a coi oce ad let A = {coi lads heads}, the for X = I{A}, X = 1 if the coi lads heads, ad X = if it lads tails. Because of this elemetary ad ituitive coi-flippig example, a Beroulli r.v. is sometimes referred to as a coi flip, where p is the probability of ladig heads. Observig the outcome of a Beroulli r.v. is sometimes called performig a Beroulli trial, or experimet. Keepig i the spirit of (1) we deote a Beroulli p r.v. by X Ber(p). 2. Biomial distributio with success probability p ad trials: If we cosecutively perform idepedet Beroulli p trials, X 1,..., X, the the total umber of successes X = X X yields the Biomial r.v. with p.m.f. { ( ) p(k) = k p k (1 p) k, if k ;, otherwise. I our coi-flippig cotext, whe cosecutively flippig the coi exactly times, p(k) deotes the probability that exactly k of the flips lad heads (ad hece exactly k lad tails). A simple computatio (utilizig X = X X ad idepedece) yields E(X) = p V ar(x) = p(1 p) M(s) = (pe s + 1 p). Keepig i the spirit of (1) we deote a biomial, p r.v. by X bi(, p). 3. geometric distributio with success probability p: The umber of idepedet Beroulli p trials required util the first success yields the geometric r.v. with p.m.f. p(k) = { p(1 p) k 1, if k 1;, otherwise. I our coi-flippig cotext, whe cosecutively flippig the coi, p(k) deotes the probability that the k th flip is the first flip to lad heads (all previous k 1 flips lad tails). The tail of X has the ice form F (k) = P (X > k) = (1 p) k, k. It ca be show that E(X) = 1 p V ar(x) = M(s) = (1 p) p 2 pe s 1 (1 p)e s. 4

5 (I fact, computig M(s) is straightforward ad ca be used to geerate the mea ad variace.) Keepig i the spirit of (1) we deote a geometric p r.v. by X geom(p). Note i passig that P (X > k) = (1 p) k, k. Remark 1.3 As a variatio o the geometric, if we chage X to deote the umber of failures before the first success, ad deote this by Y, the (sice the first flip might be a success yieldig o failures at all), the p.m.f. becomes p(k) = { p(1 p) k, if k ;, otherwise, ad p() = p. The E(Y ) = (1 p)p 1 ad V ar(y ) = (1 p)p 2. Both of the above are called the geometric distributio, ad are related by Y = X Poisso distributio with mea (ad variace) λ: With λ > a costat, X has p.m.f. { p(k) = e λ λk k!, if k ;, otherwise. The Poisso distrubutio has the iterestig property that both its mea ad variace are idetical E(X) = V ar(x) = λ. Its mgf is give by M(s) = e λ(es 1). The Poisso distributio arises as a approximatio to the biomial (, p) distributio whe is large ad p is small: Lettig λ = p, ( ) p k (1 p) k λ λk e k k!, k. Keepig i the spirit of (1) we deote a Poisso λ r.v. by Cotiuous case X P oiss(λ). 1. uiform distributio o (a, b): With a ad b costats, X has desity fuctio f(x) = { 1 b a ; if x (a, b), otherwise, c.d.f. x a b a, if x (a, b); F (x) = 1, if x b;, if x a, 5

6 ad tail A simple computatio yields b x b a, if x (a, b); F (x) =, if x b; 1, if x a. E(X) = a + b 2 (b a)2 V ar(x) = 12 M(s) = esb e sa s(b a). Whe a = ad b = 1, this is kow as the uiform distributio over the uit iterval, ad has desity f(x) = 1, x (, 1), E(X) =.5, V ar(x) = 1/12, M(s) = s 1 (e s 1). Keepig i the spirit of (1) we deote a uiform (a, b) r.v. by X uif(a, b). 2. expoetial distributio: With λ > a costat, X has desity fuctio { λe f(x) = λx, if x ;, if x <, c.d.f. ad tail A simple computatio yields F (x) = F (x) = { 1 e λx, if x ;, if x <, { e λx, if x ; 1, if x <, E(X) = 1 λ V ar(x) = 1 λ 2 M(s) = λ λ s. The expoetial is famous for havig the uique memoryless property, P (X y > x X > y) = P (X > x), x, y, i the sese that it is the uique distributio with this property. (The geometric distributio satisfies a discrete versio of this.) Keepig i the spirit of (1) we deote a expoetial λ r.v. by X exp(λ). The expoetial distributio ca be viewed as approximatig the distributio of the time util the first success whe performig a idepedet Ber(p) trial every t uits of time with p = λ t ad t very small; as t, the approximatio becomes exact. 6

7 3. ormal distributio with mea µ ad variace σ 2 : N(µ, σ 2 ): The ormal distributio is extremely importat i applicatios because of the Cetral Limit Theorem (CLT). With < µ < (the mea) ad σ 2 > (the variace) costats, X has desity fuctio f(x) = 1 σ 2π e (x µ) 2 2σ 2, < x <. This is also called the Gaussia distributio. We deote it by N(µ, σ 2 ). Whe µ = ad σ 2 = 1 it is called the stadard or uit ormal, deoted by N(, 1). If Z is N(, 1), the X = σz + µ is N(µ, σ 2 ). Similarly, if X is N(µ, σ 2 ), the Z = (x µ)/σ is N(, 1). Thus the c.d.f. F (x) ca be expressed i terms of the c.d.f. of a uit ormal Z. We therefore give the N(, 1) c.d.f. the special otatio Θ(x); ad we see that Θ(x) = P (Z x) = 1 2π x F (x) = P (X x) = P (σz + µ x) = P (Z (x µ)/σ) = Θ((x µ)/σ). e y2 2 dy, Θ(x) does ot have a closed form (e.g., a ice formula that we ca write dow ad plug ito); hece the importace of good umerical recipes for computig it, ad tables of its values. The momet geeratig fuctio of N(µ, σ 2 ) ca be show to be M(s) = e sµ+s2 σ 2 /2. Keepig i the spirit of (1) we deote a N(µ, σ 2 ) r.v. by X N(µ, σ 2 ). 4. logormal distributio: If Y is N(µ, σ 2 ), the X = e Y is a o-egative r.v. havig the logormal distributio; called so because its atural logarithm Y = l(x) yields a ormal r.v. X has desity f(x) = { 1 xσ e (l(x) µ) 2 2π 2σ 2, if x ;, if x <. Observig that E(X) ad E(X 2 ) are simply the momet geeratig fuctio of N(µ, σ 2 ) evaulated at s = 1 ad s = 2 respectively yields σ2 µ+ E(X) = e 2 V ar(x) = e 2µ+σ2 (e σ2 1). (It ca be show that M(s) = for ay s >.) 7

8 The logormal distributio plays a importat role i fiacial egieerig sice it is frequetly used to model stock prices. As with the ormal distributio, the c.d.f. does ot have a closed form, but it ca be computed from that of the ormal via P (X x) = P (Y l(x)) due to the relatio X = e Y, ad we coclude that F (x) = Θ((l(x) µ)/σ). Thus computatios for F (x) are reduced to dealig with Θ(x), the c.d.f. of N(, 1). Keepig i the spirit of (1) we deote a logormal µ, σ 2 r.v. by X logorm(µ, σ 2 ). 5. Pareto distributio: With costat a >, X has desity { ax f(x) = (1+a), if x 1;,, if x < 1, c.d.f. ad tail F (x) = F (x) = { 1 x a, if x 1;, if x 1, { x a, if x 1; 1, if x 1. (I may applicatios, a is a iteger.) A simple computatio yields E(X) = V ar(x) = (It ca be show that M(s) = for ay s >.) a, a > 1; (= otherwise) a 1 a ( a ) 2, a 2 a > 2; (= otherwise). a 1 It is easily see that E(X ) = for all a: The Pareto distributio has ifiite momets for high eough. The Pareto distributio has the importat feature that its tail F (x) = x a teds to, as x, slower tha does ay expoetial tail e λx or ay logormal tail. It is a example of a distributio with a very heavy or fat tail. Data suggests that the distributio of stock prices resembles the Pareto more tha it does the widely used logormal. Keepig i the spirit of (1) we deote a Pareto a r.v. by X P areto(a). Remark 1.4 Variatios o the Pareto distributio exist which allow the mass to start at differet locatios; F (x) = (c/(c + x)) a, x with c > ad a > costats for example. 1.4 Calculatig expected vaues by itegratig the tail Give a cotiuous o-egative radom variable X, we typically calculate, by defiitio, its expected value (also called its mea) via E(X) def = xf(x)dx, where f(x) is the desity fuctio of X. However, it is usually easier to calculate E(X) by itegratig the tail F (x): 8

9 If X is a o- Propositio 1.1 (Computig E(X) via Itegratig the Tail Method) egative radom variable, the E(X) ca be computed via Proof : Lettig E(X) = I(x) = I{X > x} def = deote the idicator fuctio for the evet {X > x}, X = X dx = F (x)dx. (9) { 1, if X > x;, if X x, I(x)dx, which is easily see by graphig the fuctio I(x) as a fuctio of x (which yields a rectagle with legth X ad height 1, thus with area X). Takig expectatios we coclude that E(X) = E{ I(x)dx}. Fially, iterchagig the order of itegral ad expected value (allowed sice everythig here is o-egative; formally this is a applicatio of Toelli s Theorem) ad recallig that E(I{B}) = P (B) for ay evet B, where here B = {X > x}, yields as was to be show. E(X) = E(I(x))dx = P (X > x)dx, Examples 1. (Expoetial distribitio:) F (x) = e λx which whe itegrated yields E(X) = e λx dx = 1 λ. Note that computig this expected value by itegratig xf(x) = xλe λx would require itegratio by parts. 2. (Computig E(mi{X, Y }) for idepedet X ad Y :) Cosider two r.v.s. X ad Y. Let Z def = mi{x, Y } (the miimum value of X ad Y ; Z = X if X Y, Z = Y if Y < X). The P (Z > z) = P (X > z, Y > z) because the miimum of X ad Y is greater tha z if ad oly if both X ad Y are greater tha z. If we also assume that X ad Y are idepedet, the P (X > z, Y > z) = P (X > z)p (Y > z), ad so (whe mi{x, Y } is o-egative) we ca compute E(Z) via E(Z) = P (Z > z)dz = P (X > z)p (Y > z)dz, a very useful result. For example, suppose X exp(1) ad Y exp(2) are idepedet. The for Z = mi{x, Y }, P (Z > z) = e z e 2z = e 3z ad we coclude that E(Z) = e 3z dz = 1/3. 9

10 (I fact what we have show here, more geerally, is that the miimum of two idepedet expotially distributed r.v.s. is itself expotially distributed with rate as the sum of the idividual rates.) 3. (Computig E{(X K) + } for a costat K :) For ay umber a, the positive part of a is defied by a + def = max{, a} (the maximum value of ad a; a + = a if a > ; otherwise). For fixed K ad ay radom variable X, with c.d.f. F (x), let Y = (X K) +, the positive part of X K. The sice Y is o-egative we ca compute E(Y ) by itegratig its tail. But for ay x, it holds that a + > x if ad oly if a > x, yieldig (X K) + > x if ad oly if (X K) > x; equivaletly if ad oly if X > x + K. We thus coclude that P (Y > x) = P (X > x + K) = F (x + K) yieldig E(Y ) = F (x + K)dx = K F (x)dx, where we chaged variables, u = x + K, to obtai the last itegral. For example suppose X uif(, 1) ad K =.5. The E(X.5) + =.5 (1 x)dx =.125 Applicatio of E{(X K) + }: Suppose that you ow a optio to buy a stock at price K = 2, at time T = 6 (moths from ow). The stock price at time T = 6 will have value X = S(T ) (radom). You will exercise the optio if ad oly if S(T ) > 2; ad do othig otherwise. (The idea beig that if S(T ) > 2, the you will buy it at price 2 ad immediately sell it at the market price S(T ) > 2.) The (S(T ) 2) + is your payoff at time T, ad E{(S(T ) 2) + } your expected payoff Computig higher momets It ca be show more geerally that the th momet of X, E(X ) = x f(x)dx, ca be computed as E(X ) = yieldig the secod momet of X whe = 2: E(X 2 ) = The proof follows from the fact that X = Discrete-time case X x 1 P (X > x)dx, 1, (1) x 1 dx = 2xP (X > x)dx. (11) x 1 I{X > x}dx. Itegratig the tail method also is valid for o-egative discrete-time radom variables; the itegral is replaced by a summatio: E(X) = P (X > k). k= 1

11 For example, if X has a geometric distributio, the F (k) = (1 p) k, k yieldig via the sum of a geometric series. E(X) = (1 p) k = 1 p, k= 1.5 Strog Law of Large Numbers ad the Cetral limit theorem (CLT) A stochastic process is a collectio of r.v.s. {X t : t T } with idex set T. If T = {, 1, 2,...} is the discrete set of itegers, the we obtai a sequece of radom variables X, X 1, X 2,... deoted by {X : } (or just {X }). I this case we refer to the value X as the state of the process at time. For example X might deote the stock price of a give stock at the ed of the th day. If time starts at = 1, the we write {X : 1} ad so o. If time is cotiuous (meaig that the idex set T = [, )) the we have a cotiuous-time stochastic process deoted by {X t : t }. A very special (but importat) case of a discrete-time stochastic process is whe the r.v.s. are idepedet ad idetically distributed (i.i.d.). I this case there are two classical ad fudametal results, the strog law of large umbers (SLLN) ad the cetral limit theorem (CLT): Theorem 1.1 (SLLN) If {X : 1} are i.i.d. with fiite mea E(X) = µ, the w.p.1., 1 X i µ,. i=1 Oe of the practical cosequeces of the SLLN is that we ca, for large eough, use the approximatio E(X) 1 X i, whe tryig to determie a apriori ukow mea E(X) = µ. For this reaso the SLLN is fudametal i Mote Carlo Simulatio. We ow state the cetral limit theorem: Theorem 1.2 (CLT) If {X : 1} are i.i.d. with fiite mea E(X) = µ ad fiite o-zero variace σ 2 = V ar(x), the Z def = 1 σ ( i=1 i=1 ) X i µ = N(, 1),, i distributio; (i other words, lim P (Z x) = Θ(x), < x <, where Θ(x) is the cdf of N(, 1).) The CLT allows us to approximate sums of i.i.d. r.v.s. edowed with ay c.d.f. F (eve if ukow) by the c.d.f. of a ormal, as log as the variace of F is fiite; it says that for sufficietly large, X i N(µ, σ 2 ). i=1 11

12 The famous ormal approximatio to the biomial distributio is but oe example, for a biomial rv ca be writte as the sum of i.i.d. Beroulli rvs, ad thus the CLT applies. Sometimes, the CLT is writte i terms of the sample average, i which case it becomes Z def = X() = 1 X j, (12) j=1 (X() µ) = N(, 1). σ If µ = ad σ 2 = 1, the the CLT simplifies to 1 X i = N(, 1), i=1 or equivaletly X() = N(, 1). The CLT yields the theoretical justificatio for the costructio of cofidece itervals, allowig oe to say, for example, that I am 95% cofidet that the true mea µ lies withi the iterval [22.2, 22.4]. We briefly review this ext. 1.6 Cofidece itervals for estimatig a ukow mea µ = E(X) I statistics, we estimate a ukow mea µ = E(X) of a distributio by collectig iid samples from the distributio, X 1,..., X ad usig as our approximatio the sample mea X() = 1 X j. (13) j=1 But how good a approximatio is this? The CLT helps aswer that questio: Lettig σ 2 = V ar(x) deote the variace of the distributio, we coclude that V ar(x()) = σ2. (14) The cetral limit theorem asserts that as, the distributio of def Z = σ (X() µ) teds to N(, 1), the uit ormal distributio. Lettig Z deote a N(, 1) rv, we coclude that for sufficietly large, Z Z i distributio. From here we obtai for ay z, P ( X() µ > z σ ) P ( Z > z) = 2P (Z > z). (We ca obtai ay value of P (Z > z) by referrig to tables, etc.) For ay α > o matter how small (such as α =.5), lettig z α/2 be such that P (Z > z α/2 ) = α/2, we thus have P ( X() µ > z α/2 σ ) α, 12

13 which ca be rewritte as P (X() z α/2 σ µ X() + z α/2 σ ) 1 α, which implies that the ukow mea µ lies withi the iterval X() ± z α/2 σ with (approximately) probability 1 α. We have thus costructed a cofidece iterval for our estimate: we say that the iterval X() ± z α/2 σ is a 1(1 α)% cofidece iterval for the mea µ. Typically, we would use (say) α =.5 i which case z α/2 = z.25 = 1.96, ad we thus obtai a 95% cofidece iterval X() ± (1.96) σ. The legth of the cofidece iterval is 2(1.96) σ which of course teds to as the sample size gets larger. The mai problem with usig such cofidece itervals is that we would ot actually kow the value of σ 2 ; it would be ukow (just as µ is). But this is ot really a problem: we istead use a estimate for it, the sample variace s 2 () defied by s 2 () = 1 1 (X j X ) 2. j=1 It ca be show that s 2 () σ 2, with probability 1, as ad that E(s 2 ()) = σ 2, 2. So, i practice we would use s() is place of σ whe costructig our cofidece itervals. For example, a 95% cofidece iterval is give by X() ± (1.96) s(). The followig recursios ca be derived; they are useful whe implemetig a simulatio requirig a cofidece iterval: X +1 = X + X +1 X, + 1 ( S+1 2 = 1 1 ) S 2 + ( + 1)(X +1 X ) Multivariate radom variables ad joit distributios With x = (x 1, x 2,..., x ) T R, the distributio of a (cotiuous) radom vector X = (X 1, X 2,..., X ) T ca be described by a joit desity fuctio f(x) = f(x 1, x 2,..., x ). If the rvs are idepedet, the this f decomposes ito the product of the idividual desity fuctios, f(x 1, x 2,..., x ) = f 1 (x 1 )f 2 (x 2 ) f (x ), where f i deotes the desity fuctio of X i. I geeral, however, correlatios exist amog the rvs ad thus such a simple product form does ot hold. The joit cdf is give by F (x) = P (X 1 x 1,..., X x ). 13

14 The momet geeratig fuctio of a radom vector is defied as follows: For θ = (θ 1,..., θ ) T M X (θ) = E(e θtx ) = E(e θ 1X 1 + +θ X ). (15) A very importat class of radom vectors appearig i applicatios are multivariate ormals (Gaussia), for the the joit distributio is completely determied by the vector of meas µ = (µ 1, µ 2,..., µ ) T = E(X) = (E(X 1 ), E(X 2 ),..., E(X )) T, ad the matrix of covariaces, Σ = (σ i,j ), where σ i,j = Cov(X i, X j ) = E(X i X j ) E(X i )E(X j ), with < σ ii = σi 2 = V ar(x i ), i, j {1, 2..., }. We deote such a ormal vector by X N(µ, Σ). Such a Σ is symmetric, ad positive semidefiite, Σ T = Σ, x T Σx, x R, ad ay matrix with those two properties defies a covariace matrix for a multivariate ormal distributio. Positive semidefiite is equivalet to all eigevalues of Σ beig oegative. The momet geeratig fuctio (15) has a elegat form aalogous to the oedimesioal ormal case: M X (θ) = E(e θtx ) = e µt θ+ 1 2 θtσθ. (16) A very ice feature of a Gaussia vector X is that it ca always be expressed as a liear trasformatio of iid N(, 1) rvs. If Z = (Z 1, Z 2,..., Z ) T are iid N(, 1), the there exists a liear mappig (matrix) A (from R to R ), such that X = AZ + µ. (17) I this case Σ = AA T. Coversely, ay liear trasformatio of a Gaussia vector is yet agai Gaussia: If X N(µ, Σ) ad B is a m matrix, the Y = BX N(Bµ, BΣB T ) (18) is a m dimesioal Gaussia where m 1 ca be ay iteger. For purposes of costructig a desired X N(µ, Σ) usig (17) a solutio A to Σ = AA T is ot uique (i geeral), but it is easy to fid oe such solutio by diagoalizatio: Because Σ is symmetric with real elemets, it ca be re-writte as Σ = UDU T, where D is the diagoal matrix of the eigevalues, ad U is a orthogoal matrix (U T = U 1 ) with colums as the associated eigevectors. Because Σ is also positive semidefiite, all the eigevalues must be o-egative yieldig the existece of D ad thus a solutio A = U D. (19) Thus fidig a A reduces to fidig the U ad the D. There also exists a lower triagular solutio to Σ = AA T called the Cholesky decompositio of Σ. Such a decompositio is more attractive computatioally (for algebraically computig X from Z) sice it requires less additios ad multiplicatios alog the way. If the matrix Σ has full rak (equivaletly, Σ is positive defiite (all eigevalues are positive) as opposed to oly beig positive semidefiite), the the Cholesky decompositio ca be easily computed iteratively by solvig for it directly: Assumig A is lower triagular, solve the set of equatios geerated whe settig Σ = AA T. We will study this i more detail later whe we lear how to efficietly simulate multivariate ormal rvs. 14

15 1.8 Practice problems 1. Each of the followig o-egative radom variables X has a cotiuous distributio. Calculate the expected value E(X) i two ways: Directly via E(X) = xf(x)dx ad by itegratig the tail E(X) = P (X > x)dx. (a) Expoetial distributio with rate λ. (b) Uiform distributio o (1, 3) (c) Pareto distributio with a = (Cotiued) Now repeat the above for calculatig the secod momet E(X 2 ) of each r.v. i two ways: Directly via E(X 2 ) = x 2 f(x)dx ad by use of (11). 3. (Cotiued) Usig your aswers from above, give the variace V ar(x) of each radom variable, V ar(x) = E(X 2 ) (E(X)) Compute E{(X 2) + } whe X has a expoetial distributio with rate λ = Let X have a expoetial distributio with rate λ. Show that X has the memoryless property: P (X y > x X > y) = P (X > x), x, y. Now let X have a geometric distributio with success p. Show that X has the memoryless property: P (X m > k X > m) = P (X > k), k, m. (k ad m are itegers oly.) 6. The joit desity of X > ad Y > is give by Show that E(X Y = y) = y. f(x, y) = e x/y e y, x >, y >. y 7. Let X have a uiform distributio o the iterval (, 1). Fid E(X X <.5). 8. Suppose that X has a expoetial distributio with rate 1: P (X x) = 1 e x, x. For λ >, show that Y = (λ 1 X) 2 has a Weibull distributio, P (Y x) = 1 e λ x, x. From this fact, ad the algorithm give i Lecture 1 for geeratig (simulatig) a expoetial rv from a uit uiform U rv, give a algorithm for simulatig such a Y that uses oly oe U. 9. A ma is trapped i a cave. There are three exits, 1, 2, 3. If he chooses 1, the after a 2 day jourey he eds up back i the cave. If he chooses 2, the after a 4 day jourey he eds up back i the cave. If he chooses 3, the after a 5 day jourey he is outside ad free. He is equally likely to choose ay oe of the three exits, ad each time (if ay) he returs back i the cave he is agai, idepedet of his past mistakes, equally likely to choose ay oe of the three exits (it is dark so he ca t see which oe he chose before). Let N deote the total umber of attempts (exit choosigs) he makes util becomig free. Let X deote the total umber of days util he is free. 15

16 (a) Compute P (N = k), k 1. What ame is give to this famous distributio? (b) Compute E(N). (c) By usig the method of coditioig (coditio o the exit first chose), compute E(X). Hit: Let D deote the first exit chose. The E(X D = 1) = 2 + E(X), ad so o, where P (D = i) = 1/3, i = 1, 2, 3. Moreover E(X) = 3 i=1 E(X D = i)p (D = i). 16