Probability Generating Functions

Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence {a i : i = 0,, 2, } of real numbers: the numbers can be parcelled up in several kinds of generating functions The ordinary generating function of the sequence is defined as G(s) = a i s i for those values of the parameter s for which the sum converges For a given sequence, there exists a radius of convergence R( 0) such that the sum converges absolutely if s < R and diverges if s > R G(s) may be differentiated or integrated term by term any number of times when s < R For many well-defined sequences, G(s) can be written in closed form, and the individual numbers in the sequence can be recovered either by series expansion or by taking derivatives i=0 32 Definitions and Properties Consider a count rv X, ie a discrete rv taking non-negative values Write p k = P(X = k), k = 0,, 2, (3) (if X takes a finite number of values, we simply attach zero probabilities to those values which cannot occur) The probability generating function (PGF) of X is defined as G X (s) = p k s k = E(s X ) (32) Note that G X () =, so the series converges absolutely for s Also G X (0) = p 0 For some of the more common distributions, the PGFs are as follows: (i) Constant rv if p c =, p k = 0, k c, then G X (s) = E(s X ) = s c (33)

(ii) Bernoulli rv if p = p, p 0 = p = q, p k = 0, k 0 or, then G X (s) = E(s X ) = q + ps (34) (iii) Geometric rv if p k = pq k, k =, 2, ; q = p, then G X (s) = ps qs (iv) Binomial rv if X Bin(n, p), then if s < q (see HW Sheet 4) (35) G X (s) = (q + ps) n, (q = p) (see HW Sheet 4) (36) (v) Poisson rv if X Poisson(λ), then G X (s) = k! λk e λ s k = e λ(s ) (37) (vi) Negative binomial rv if X NegBin(n, p), then ( ) ( ) k ps n G X (s) = p n q k n s k = if s < q and p + q = (38) n qs Uniqueness Theorem If X and Y have PGFs G X and G Y respectively, then G X (s) = G Y (s) for all s (a) iff P(X = k) = P(Y = k) for k = 0,, (b) ie if and only if X and Y have the same probability distribution Proof We need only prove that (a) implies (b) The radii of convergence of G X and G Y are, so they have unique power series expansions about the origin: G X (s) = G Y (s) = s k P(X = k) s k P(Y = k) If G X = G Y, these two power series have identical coefficients ps Example: If X has PGF with q = p, then we can conclude that ( qs) X Geometric(p) Given a function A(s) which is known to be a PGF of a count rv X, we can obtain p k = P(X = k) either by expanding A(s) in a power series in s and setting p k = coefficient of s k ; or by differentiating A(s) k times with respect to s and setting s = 0

We can extend the definition of PGF to functions of X The PGF of Y = H(X) is ( G Y (s) = G H(X) (s) = E s H(X)) = k P(X = k)s H(k) (39) If H is fairly simple, it may be possible to express G Y (s) in terms of G X (s) Example: Let Y = a + bx Then G Y (s) = E(s Y ) = E(s a+bx ) = s a E[(s b ) X ] = s a G X (s b ) (30) 33 Moments Theorem Let X be a count rv and G (r) X () the rth derivative of its PGF G X(s) at s = Then Proof (informal) G (r) X () = E[X(X )(X r + )] (3) G (r) dr X (s) = ds r [G X(s)] = dr ds r = k [ ] p k s k k p k k(k )(k r + )s k r (assuming the interchange of d r ds r and k is justified) This series is convergent for s, so G (r) X () = E[X(X )(X r + )], r In particular: and so G () X () (or G X()) = E(X) (32) G (2) X () (or G X ()) = E[X(X )] = E(X 2 ) E(X) = Var(X) + [E(X)] 2 E(X) Var(X) = G (2) [ X () G () ] 2 X () () + G X () (33) Example: If X Poisson(λ), then G X (s) = e λ(s ) ; G () X (s) = λeλ(s ) so E(X) = G () X () = λe0 = λ G (2) X (s) = λ2 e λ(s ) so Var(X) = λ 2 λ 2 + λ = λ

34 Sums of independent random variables Theorem Let X and Y be independent count rvs with PGFs G X (s) and G Y (s) respectively, and let Z = X + Y Then Proof Corollary G Z (s) = G X+Y (s) = G X (s)g Y (s) (34) G Z (s) = E(s Z ) = E(s X+Y ) = E(s X )E(s Y ) (independence) = G X (s)g Y (s) If X,, X n are independent count rvs with PGFs G X (s),, G Xn (s) respectively (and n is a known integer), then Example 3 G X ++X n (s) = G X (s)g Xn (s) (35) Find the distribution of the sum of n independent rvs X i, i =, n, where X i Poisson(λ i ) Solution So This is the PGF of a Poisson rv, ie Example 32 G Xi (s) = e λ i(s ) G X +X 2 + +X n (s) = n e λ i(s ) i= = e (λ + +λ n)(s ) n n X i Poisson( λ i ) (36) i= i= In a sequence of n independent Bernoulli trials Let {, if the ith trial yields a success (probability p) I i = 0, if the ith trial yields a failure (probability q = p) n Let X = I i = number of successes in n trials What is the probability distribution of X? i= Solution Since the trials are independent, the rvs I, I n are independent So G X (s) = G I G I2 G In (s) But G Ii (s) = q + ps, Then ie X Bin(n, p) i =, n So G X (s) = (q + ps) n = n x=0 ( ) n (ps) x q n x x P(X = x) = coefficient of s x in G X (s) = ( n x) p x q n x, x = 0,, n

35 Sum of a random number of independent rvs Theorem Let N, X, X 2, be independent count rvs If the {X i } are identically distributed, each with PGF G X, then has PGF S N = X + + X N (37a) G SN = G N (G X (s)) (37b) (Note: We adopt the convention that X + + X N = 0 for N = 0) This is an example of the process known as compounding with respect to a parameter Proof We have ( G SN (s) = E = E = = = n=0 n=0 n=0 n=0 ) s S N ( ) s S N N = n P(N = n) (conditioning on N) ) E (s Sn P(N = n) G X + +X n (s)p(n = n) [G X (s)] n P(N = n) (using Corollary in previous section) = G N (G X (s)) by definition of G N Corollaries E(S N ) = E(N)E(X) (38) Proof d ds [G S N (s)] = d ds [G N (G X (s))] = dg N (u) du where u = G X (s) du ds Setting s =, (so that u = G X () = ), we get E(S N ) = Similarly it can be deduced that [ G () ] [ N () G () ] X () = E(N)E(X) 2 Var(S N ) = E(N)Var(X) + Var(N) [E(X)] 2 (39) (prove!)

Example 33 The Poisson Hen A hen lays N eggs, where N Poisson(λ) Each egg hatches with probability p, independently of the other eggs Find the probability distribution of the number of chicks, Z Solution We have Z = X + + X N, where X, are independent Bernoulli rvs with parameter p Then G N (s) = e λ(s ), G X (s) = q + ps So G Z (s) = G N (G X (s)) = e λp(s ) the PGF of a Poisson rv, ie Z Poisson(λp) 36 *Using GFs to solve recurrence relations [NOTE: Not required for examination purposes] Instead of appealing to the theory of difference relations when attempting to solve the recurrence relations which arise in the course of solving problems by conditioning, one can often convert the recurrence relation into a linear differential equation for a generating function, to be solved subject to appropriate boundary conditions Example 34 The Matching Problem (yet again) At the end of Chapter [Example (0)], the recurrence relation p n = n n p n + n p n 2, n 3 was derived for the probability p n that no matches occur in an n-card matching problem Solve this using an appropriate generating function Solution Multiply through by ns n and sum over all suitable values of n to obtain Introducing the GF ns n p n = s (n )s n 2 p n + s s n 2 p n 2 n=3 n=3 n=3 G(s) = p n s n n= (NB: this is not a PGF, since {p n : n } is not a probability distribution), we can rewrite this as G (s) p 2p 2 s = s[g (s) p ] + sg(s) ie ( s)g (s) = sg(s) + s (since p = 0, p 2 = 2 ) This first order differential equation now has to be solved subject to the boundary condition The result is G(0) = 0 G(s) = ( s) e s Now expand G(s) as a power series in s, and extract the coefficient of s n : this yields p n = + ( )! + ( )n (n )! + ( )n, n n! the result obtained previously

37 Branching Processes 37 Definitions Consider a hypothetical organism which lives for exactly one time unit and then dies in the process of giving birth to a family of similar organisms We assume that: (i) the family sizes are independent rvs, each taking the values 0,,2,; (ii) the family sizes are identically distributed rvs, the number of children in a family, C, having the distribution P(C = k) = p k, k = 0,, 2, (320) The evolution of the total population as time proceeds is termed a branching process: this provides a simple model for bacterial growth, spread of family names, etc Let X n = number of organisms born at time n (ie, the size of the nth generation) (32) The evolution of the population is described by the sequence of rvs X 0, X, X 2, (an example of a stochastic process of which more later) We assume that X 0 =, ie we start with precisely one organism A typical example can be shown as a family tree: 0 X 0 = Generation 2 X = 3 X 2 = 6 3 X 3 = 9 We can use PGFs to investigate this process 372 Evolution of the population Let G(s) be the PGF of C: G(s) = P(C = k)s k, (322) and G n (s) the PGF of X n : G n (s) = P(X n = x)s x (323) x=0 Now G 0 (s) = s, since P(X 0 = ) = ; P(X 0 = x) = 0 for x, and G (s) = G(s) Also X n = C + C 2 + + C Xn, (324) where C i is the size of the family produced by the ith member of the (n )th generation

So, since X n is the sum of a random number of independent and identically distributed (iid) rvs, we have G n (s) = G n (G(s)) for n = 2, 3, (325) and this is also true for n = Iterating this result, we have ie, G n is the nth iterate of G G n (s) = G n (G(s)) = G n 2 (G(G(s))) = = G (G(G((s)))) = G(G(G((s)))), n = 0,, 2, Now ie E(X n ) = G n () = G n (G())G () = G n ()G () [since G() = ] where µ = E(C) is the mean family size Hence It follows that E(X n ) = E(X n )µ (326) E(X n ) = µe(x n ) = µ 2 E(X n 2 ) = = µ n E(X 0 ) = µ n 0, if µ < E(X n ), if µ = (critical value), if µ > (327) It can be shown similarly that nσ 2, if µ = Var(X n ) = σ 2 µ n µn µ, if µ (328) Example 35 Investigate a branching process in which C has the modified geometric distribution p k = pq k, k = 0,, 2, ; 0 < p = q <, with p q [NB] Solution The PGF of C is G(s) = pq k s k = p qs if s < q We now need to solve the functional equations G n (s) = G n (G(s))

First, if s, G (s) = G(s) = (remember we have assumed that p q) Then We therefore conjecture that p qs = p (q p) (q 2 p 2 ) qs(q p) p G 2 (s) = G(G(s)) = qp qs p( qs) = qp qs = p (q2 p 2 ) qs(q p) (q 3 p 3 ) qs(q 2 p 2 ) G n (s) = p (qn p n ) qs(q n p n ) (q n+ p n+ ) qs(q n p n ) for n =, 2, and s, and this result can be proved by induction on n We could derive the entire probability distribution of X n by expanding the rhs as a power series in s, but the result is rather complicated In particular, however, q n p n P(X n = 0) = G n (0) = p q n+ p n+ = µn µ n+, where µ = q/p is the mean family size It follows that ultimate extinction is certain if µ < and less than certain if µ > Separate consideration of the case p = q = 2 (see homework) shows that ultimate extinction is also certain when µ = We now proceed to show that these conclusions are valid for all family size distributions 373 The Probability of Extinction The probability that the process is extinct by the nth generation is e n = P(X n = 0) (329) Now e n, and e n e n+ (since X n = 0 implies that X n+ = 0) - ie {e n } is a bounded monotonic sequence So exists and is called the probability of ultimate extinction Theorem e is the smallest non-negative root of the equation x = G(x) Proof Note that e n = P(X n = 0) = G n (0) Now e = lim n e n (330) G n (s) = G n (G(s)) = = G(G(s))) = G(G n (s))

Set s = 0 Then e n = G n (0) = G(e n ), n =, 2, with boundary condition e 0 = 0 In the limit n we have e = G(e) Now let η be any non-negative root of the equation x = G(x) G is non-decreasing on the interval [0, ] (since it has non-negative coefficients), and so and by induction So e = G(e 0 ) = G(0) G(η) = η; e 2 = G(e ) G(η) = η [using previous line] e n η, n =, 2, e = lim n e n η It follows that e is the smallest non-negative root Theorem 2 e = if and only if µ [Note: we rule out the special case p = ; p k = 0 for k, when µ = but e = 0] Proof We can suppose that p 0 > 0 (since otherwise e = 0 and µ > ) Now on the interval [0, ], G is (i) continuous (since radius of convergence ); (ii) non-decreasing (since G (x) = k kp k x k 0); (iii) convex (since G (x) = k k(k )p k x k 2 0) It follows that in [0, ] the line y = x has either or 2 intersections with the curve y = G(x),as shown below: y y y=g(x) y=x e y=g(x) y=x e (a) x (b) x Since G () = µ, it follows that The curve y = G(x) and the line y = x, in the two cases when (a) G () > and (b) G () e = if and only if µ

Example 36 Find the probability of ultimate extinction when C has the modified geometric distribution p k = pq k, k = 0,, 2, ; 0 < p = q < (the case p = q is now permitted - cf Example 35) Solution The PGF of C is G(s) = p qs, s < q, and e is the smallest non-negative root of The roots are So G(x) = p qx = x ± (2p ) 2( p) if p < 2 (ie µ = q/p > ), e = p/q(= µ ); if p 2 (ie µ ), e = in agreement with our earlier discussion in Example 35 [Note: in the above discussion, the continuity of probability measures (see Notes below eqn (0)) has been invoked more than once without comment]