MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 14 10/27/2008 MOMENT GENERATING FUNCTIONS

Transcription

1 MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 14 10/27/2008 MOMENT GENERATING FUNCTIONS Contents 1. Moment generating functions 2. Sum of a ranom number of ranom variables 3. Transforms associate with joint istributions Moment generating functions, an their close relatives (probability generating functions an characteristic functions) provie an alternative way of representing a probability istribution by means of a certain function of a single variable. These functions turn out to be useful in many ifferent ways: (a) They provie an easy way of calculating the moments of a istribution. (b) They provie some powerful tools for aressing certain counting an combinatorial problems. (c) They provie an easy way of characterizing the istribution of the sum of inepenent ranom variables. () They provie tools for ealing with the istribution of the sum of a ranom number of inepenent ranom variables. (e) They play a central role in the stuy of branching processes. (f) They play a key role in large eviations theory, that is, in stuying the asymptotics of tail probabilities of the form P(X c), when c is a large number. (g) They provie a brige between complex analysis an probability, so that complex analysis methos can be brought to bear on probability problems. (h) They provie powerful tools for proving limit theorems, such as laws of large numbers an the central limit theorem. 1

2 1 MOMENT GENERATING FUNCTIONS 1.1 Definition Definition 1. The moment generating function associate with a ranom variable X is a function M X : R [0, ] efine by M X (s) = E[e sx ]. The omain D X of M X is efine as the set D X = {s M X (s) < }. If X is a iscrete ranom variable, with PMF p X, then M X (s) = e sx p X (x). If X is a continuous ranom variable with PDF f X, then M X (s) = e sx f X (x) x. x Note that this is essentially the same as the efinition of the Laplace transform of a function f X, except that we are using s instea of s in the exponent. 1.2 The omain of the moment generating function Note that 0 D X, because M X (0) = E[e 0X ] = 1. For a iscrete ranom variable that takes only a finite number of ifferent values, we have D X = R. For example, if X takes the values 1, 2, an 3, with probabilities 1/2, 1/3, an 1/6, respectively, then M X (s) = e s + e 2s + e 3s, (1) which is finite for every s R. On the other han, for the Cauchy istribution, f X (x) = 1/(π(1 + x 2 )), for all x, it is easily seen that M X (s) =, for all s = 0. In general, D X is an interval (possibly infinite or semi-infinite) that contains zero. Exercise 1. Suppose that M X (s) < for some s > 0. Show that M X (t) < for all t [0, s]. Similarly, suppose that M X (s) < for some s < 0. Show that M X (t) < for all t [s, 0]. 2

3 Exercise 2. Suppose that log P(X > x) lim sup ν < 0. x x Establish that M X (s) < for all s [0, ν). 1.3 Inversion of transforms By inspection of the formula for M X (s) in Eq. (1), it is clear that the istribution of X is reaily etermine. The various powers e sx inicate the possible values of the ranom variable X, an the associate coefficients provie the corresponing probabilities. At the other extreme, if we are tol that M X (s) = for every s = 0, this is certainly not enough information to etermine the istribution of X. On this subject, there is the following funamental result. It is intimately relate to the inversion properties of Laplace transforms. Its proof requires sophisticate analytical machinery an is omitte. Theorem 1. Inversion theorem (a) Suppose that M X (s) is finite for all s in an interval of the form [ a, a], where a is a positive number. Then, M X etermines uniquely the CDF of the ranom variable X. (b) If M X (s) = M Y (s) <, for all s [ a, a], where a is a positive number, then the ranom variables X an Y have the same CDF. There are explicit formulas that allow us to recover the PMF or PDF of a ranom variable starting from the associate transform, but they are quite ifficult to use (e.g., involving contour integrals ). In practice, transforms are usually inverte by pattern matching, base on tables of known istribution-transform pairs. 1.4 Moment generating properties There is a reason why M X is calle a moment generating function. Let us consier the erivatives of M X at zero. Assuming for a moment we can interchange 3

4 the orer of integration an ifferentiation, we obtain M X (s) = E[e sx ] = E[Xe sx ] = E[X], s s=0 s s=0 s=0 m M X (s) = m E[e sx sx ] = E[X m e ] = E[X m ] s m s=0 s m s=0 s=0 Thus, knowlege of the transform M X allows for an easy calculation of the moments of a ranom variable X. Justifying the interchange of the expectation an the ifferentiation oes require some work. The steps are outline in the following exercise. For simplicity, we restrict to the case of nonnegative ranom variables. Exercise 3. Suppose that X is a nonnegative ranom variable an that M X (s) < for all s (, a], where a is a positive number. (a) Show that E[X k ] <, for every k. (b) Show that E[X k e sx ] <, for every s < a. (c) Show that (e hx 1)/h Xe hx. () Use the DCT to argue that hx e 1 E[e hx ] 1 E[X] = E lim = lim. h 0 h h 0 h 1.5 The probability generating function For iscrete ranom variables, the following probability generating function is sometimes useful. It is efine by g X (s) = E[s X ], with s usually restricte to positive values. It is of course closely relate to the moment generating function in that, for s > 0, we have g X (s) = M X (log s). One ifference is that when X is a positive ranom variable, we can efine g X (s), as well as its erivatives, for s = 0. So, suppose that X has a PMF p X (m), for m = 1,.... Then, g X (s) = s m p X (m), m=1 resulting in m g X (s) = m! p X (m). s m s=0 (The interchange of the summation an the ifferentiation nees justification, but is inee legitimate for small s.) Thus, we can use g X to easily recover the PMF p X, when X is a positive integer ranom variable. 4

5 1.6 Examples Example : X = Exp(λ). Then, λ, s < λ; M X (s) = e sx λe λx x = λ s 0, otherwise. Example : X = Ge(p) es p M X (s) = e sm p(1 p) m 1 1 (1 p)e, e s < 1/(1 p); s, otherwise. m=1 In this case, we also fin g X (s) = ps/(1 (1 p)s), s < 1/(1 p) an g X (s) =, otherwise. Example : X = N(0, 1). Then, 1 x 2 M X (s) = exp(sx) exp( )x 2π 2 = exp(s2 /2) exp( x2 + 2sx s 2 )x 2π 2 = exp(s 2 /2). 1.7 Properties of moment generating functions We recor some useful properties of moment generating functions. Theorem 2. (a) If Y = ax + b, then M Y (s) = e sb M X (as). (b) If X an Y are inepenent, then M X+Y (s) = M X (s)m Y (s). (c) Let X an Y be inepenent ranom variables. Let Z be equal to X, with probability p, an equal to Y, with probability 1 p. Then, M Z (s) = pm X (s) + (1 p)m Y (s). Proof: For part (a), we have M X (ax + b) = E[exp(saX + sb)] = exp(sb)e[exp(sax)] = exp(sb)m X (as). 5

6 For part (b), we have M X+Y (s) = E[exp(sX + sy )] = E[exp(sX)]E[exp(sY )] = M X (s)m Y (s). For part (c), by conitioning on the ranom choice between X an Y, we have M Z (s) = E[e sz ] = pe[e sx ] + (1 p)e[e sy ] = pm X (s) + (1 p)m Y (s). Example : (Normal ranom variables) (a) Let X be a stanar normal ranom variable, an let Y = σx + µ, which we know to have a N (µ, σ 2 ) istribution. We then fin that M Y (s) = exp(sµ s2 σ 2 ). (b) Let X = N(µ1, σ 2 1 ) an Y = N(µ 2, σ 2 2 ). Then, 1 M X+Y (s) = exp( s(µ 1 + µ 2 ) + s 2 2 (σ σ 2 2 ). Using the inversion property of transforms, we conclue that X + Y = N (µ µ 2, σ 1 + σ 2 2 ), thus corroborating a result we first obtaine using convolutions. 2 SUM OF A RANDOM NUMBER OF INDEPENDENT RANDOM VARI ABLES Let X 1, X 2,... be a sequence of i.i.. ranom variables, with mean µ an variance σ 2. Let N be another inepenent ranom variable that takes nonnegative integer values. Let Y = N i=1 X i. Let us erive the mean, variance, an moment generating function of Y. We have E[Y ] = E[E[Y N]] = E[Nµ] = E[N]E[X]. Furthermore, using the law of total variance, var(y ) = E var(y N) + var E[Y N] Finally, note that = E[Nσ 2 ] + var(nµ) = E[N]σ 2 + µ 2 var(n). E[exp(sY ) N = n] = M n (s) = exp(n log M X (s)), X 6

7 implying that M Y (s) = exp(n log M X (s))p(n = n) = M N (log M X (s)). n=1 The reaer is encourage to take the erivative of the above expression, an evaluate it at s = 0, to recover the formula E[Y ] = E[N]E[X]. Example : Suppose that each X i is exponentially istribute, with parameter λ, an that N is geometrically istribute, with parameter p (0, 1). We fin that e log M X (s) p pλ/(λ s) λp M Y (s) = = = 1 e log M X (s) (1 p) 1 λ(1 p)/(λ s) λp s which we recognize as a moment generating function of an exponential ranom variable with parameter λp. Using the inversion theorem, we conclue that Y is exponentially istribute. In view of the fact that the sum of a fixe number of exponential ranom variables is far from exponential, this result is rather surprising. An intuitive explanation will be provie later in terms of the Poisson process. 3 TRANSFORMS ASSOCIATED WITH JOINT DISTRIBUTIONS If two ranom variables X an Y are escribe by some joint istribution (e.g., a joint PDF), then each one is associate with a transform M X (s) or M Y (s). These are the transforms of the marginal istributions an o not convey information on the epenence between the two ranom variables. Such information is containe in a multivariate transform, which we now efine. Consier n ranom variables X 1,..., X n relate to the same experiment. Let s 1,..., s n be real parameters. The associate multivariate transform is a function of these n parameters an is efine by M X1,...,X n (s 1,..., s n ) = E e s 1X 1 + +s nx n. The inversion property of transforms iscusse earlier extens to the multivariate case. That is, if Y 1,..., Y n is another set of ranom variables an M X1,...,X n (s 1,..., s n ), M Y1,...,Y n (s 1,..., s n ) are the same functions of s 1,..., s n, in a neighborhoo of the origin, then the joint istribution of X 1,..., X n is the same as the joint istribution of Y 1,..., Y n. 7

8 Example : (a) Consier two ranom variables X an Y. Their joint transform is M X,Y (s, t) = E[e sx e ty ] = E[e sx+ty ] = M Z (1), where Z = sx+ty. Thus, calculating a multivariate transform essentially amounts to calculating the univariate transform associate with a single ranom variable that is a linear combination of the original ranom variables. (b) If X an Y are inepenent, then M X,Y (s, t) = M X (s)m Y (t). 8

9 MIT OpenCourseWare J / J Funamentals of Probability Fall 2008 For information about citing these materials or our Terms of Use, visit: